Question: Represent the text to find a very short summary (1-2 sentences max).

Question: We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained using adaptive gradient descent techniques with L2 regularization or weight decay. Through an extensive empirical study we hypothesize the mechanism be hind the sparsification process. We find that the interplay of various phenomena influences the strength of L2 and weight decay regularizers, leading the supposedly non sparsity inducing regularizers to induce filter sparsity. In this workshop article we summarize some of our key findings and experiments, and present additional on modern network architectures such as ResNet-50. In this article we discuss the findings from BID7 regarding filter level sparsity which emerges in certain types of feedforward convolutional neural networks. Filter refers to the weights and the nonlinearity associated with a particular feature, acting together as a unit. We use filter and feature interchangeably throughout the document. We particularly focus on presenting evidence for the implicit sparsity, our experimentally backed hypotheses regarding the cause of the sparsity, and discuss the possible role such implicit sparsification plays in the adaptive vs vanilla (m)SGD generalization debate. For implications on neural network speed up, refer to the original paper BID7.In networks which employ Batch Normalization and ReLU activation, after training, certain filters are observed to not activate for any input. Importantly, the sparsity emerges in the presence of regularizers such as L2 and weight decay (WD) which are in general understood to be non sparsity inducing, and the sparsity vanishes when regularization is 1 Max Planck Institute For Informatics, Saarbrücken, Germany 2 Saarland Informatics Campus, Germany 3 Ulsan National Institute of Science and Technology, South Korea. th International Conference on. removed. We experimentally observe the following:• The sparsity is much higher when using adaptive flavors of SGD vs. (m)SGD. The sparsity exists even with leaky ReLU.• Adaptive methods see higher sparsity with L2 regularization than with WD. No sparsity emerges in the absence of regularization.• In addition to the regularizers, the extent of the emergent sparsity is also influenced by hyperparameters seemingly unrelated to regularization. The sparsity decreases with increasing mini-batch size, decreasing network size and increasing task difficulty.• The primary hypothesis that we put forward is that selective features 1 see a disproportionately higher amount of regularization than non-selective ones. This consistently explains how unrelated parameters such as mini-batch size, network size, and task difficulty indirectly impact sparsity by affecting feature selectivity.• A secondary hypothesis to explain the higher sparsity observed with adaptive methods is that Adam (and possibly other) adaptive approaches learn more selective features. Though threre is evidence of highly selective features with Adam, this requires further study.• Synthetic experiments show that the interaction of L2 regularizer with the update equation in adaptive methods causes stronger regularization than WD. This can explain the discrepancy in sparsity between L2 and WD.Quantifying Feature Sparsity: Feature sparsity can be measured by per-feature activation and by per-feature scale. For sparsity by activation, the absolute activations for each feature are max pooled over the entire feature plane. If the value is less than 10 −12 over the entire training corpus, the feature is inactive. For sparsity by scale, we consider the scale γ of the learned affine transform in the Batch Norm layer. We consider a feature inactive if |γ| for the feature is less than 10 −3. Explicitly zeroing the features thus marked inactive does not affect the test error, which ensures the validity of our chosen thresholds. The thresholds chosen are purposefully conservative, and comparable levels of sparsity are observed for a higher feature activation threshold of 10 −4, and a higher |γ| threshold of 10 −2. Figure 1. BasicNet: Structure of the basic convolution network studied in this paper. We refer to the convolution layers as C1-7. Preliminary Experiments: We use a 7-layer convolutional network with 2 fully connected layers as shown in Figure 1. We refer to this network as BasicNet in the rest of the document. For the basic experiments on CIFAR-10/100, we use a variety of gradient descent approaches, a mini-batch size of 40, with a method specific base learning rate for 250 epochs which is scaled down by 10 for an additional 75 epochs. The base learning rates and other hyperparameters are as follows: Adam (1e-3, β 1 =0.9, β 2 =0.99, =1e-8), Adadelta (1.0, ρ=0.9, =1e-6), SGD (0.1, mom.=0.9), Adagrad (1e-2), AMSGrad (1e-3), AdaMax (2e-3), RMSProp (1e-3). We study the effect of varying the amount and type of regularization 2 on the extent of sparsity and test error in TAB0. It shows significant convolutional filter sparsity emerges with adaptive gradient descent methods when combined with L2 regularization. The extent of sparsity is reduced when using Weight Decay instead, and absent entirely in the case of SGD with moderate levels of regularization. Table 2 shows that using leaky ReLU does not prevent sparsification. The emergence of sparsity is not an isolated phenomenon specifc to CIFAR-10/100 and BasicNet. We show in tables 3, 4, and 5 that sparsity manifests in VGG-11/16 ( ), and ResNet-50 (BID0) on ImageNet and Tiny-ImageNet. ResNet-50 shows a significantly higher overall filter sparsity than nonresidual VGG networks. We see in TAB3, 3, 4, and 5 that decreasing the minibatch size (while maintaining the same number of iterations) leads to increased sparsity across network architectures and datasets. Table 4. Effect of different mini-batch sizes on sparsity (by γ) in VGG-11, trained on ImageNet. Same network structure employed as (Norm outputs {x i} N i=1 of a particular convolutional kernel, where N is the size of the training corpus, due to the use of ReLU, a gradient is only seen for those datapoints for whichx i > −β/γ. Both SGD and Adam (L2: 1e-5) learn positive γs for layer C6, however βs are negative for Adam, while for SGD some of the biases are positive. This implies that all features learned for Adam (L2: 1e-5) in this layer activate for ≤ half the activations from the training corpus, while SGD has a significant number of features activate for more than half of the training corpus, i.e., Adam learns more selective features in this layer. Features which activate only for a small subset of the training corpus, and consequently see gradient updates from the main objective less frequently, continue to be acted upon by the regularizer. If the regularization is strong enough (Adam with L2: 1e-4 in FIG2), or the gradient updates infrequent enough (feature too selective), the feature may be pruned away entirely. The propensity of later layers to learn more selective features with Adam would explain the higher degree of sparsity seen for later layers as compared to SGD. Understanding the reasons for emergence of higher feature selectivity in Adam than SGD, and verifying if other adaptive gradient descent flavours also exhibit higher feature selectivity remains open for future investigation. Quantifying Feature Selectivity: Similar to feature sparsity by activation, we apply max pooling to a feature's absolute activations over the entire feature plane. For a particular feature, we consider these pooled activations over the entire training corpus to quantify feature selectivity. See the original paper BID7 for a detailed discussion. Unlike the selectivity metrics employed in literature BID9, ours is class agnostic, and provides preliminary quantitative evidence that Adam (and perhaps other adaptive gradient descent methods) learn more selective features than (m)SGD, which consequently see a higher relative degree of regularization. Interaction of L2 Regularizer with Adam: Next, we consider the role of the L2 regularizer vs. weight decay. In the original paper we study the behaviour of L2 regularization in the low gradient regime for different optimizers through synthetic experiments and find that coupling of L2 regularization with certain adaptive gradient update equations yields a faster decay than weight decay, or L2 regularization with SGD, even for smaller regularizer values. This is an additional source of regularization disparity between parameters which see frequent updates and those which don't see frequent updates or see lower magnitude gradients. It manifests for certain adaptive gradient descent approaches. Task'Difficulty' Dependence: As per the hypothesis developed thus far, as the task becomes more difficult, for a given network capacity, we expect the fraction of features pruned to decrease corresponding to a decrease in selectivity of the learned features BID13. Since the task difficulty cannot be cleanly decoupled from the number of classes, we devise a synthetic experiment based on grayscale renderings of 30 object classes from ObjectNet3D BID11. We construct 2 identical sets of ≈ 50k 64×64 pixel renderings, one with a clean (BG) and the other with a cluttered BG. We train BasicNet with a mini-batch size of 40, and see that as expected there is a much higher sparsity (70%) with the clean BG set than with the more difficult cluttered set (57%). See the original paper BID7 for representative images and a list of the object classes selected. BID12 ) employ explicit filter sparsification heuristics that make use of the learned scale parameter γ in Batch Norm for enforcing sparsity on the filters. BID12 argue that BatchNorm makes feature importance less susceptible to scaling reparameterization, and the learned scale parameters (γ) can be used as indicators of feature importance. We thus adopt γ as the criterion for studying implicit feature pruning. Morcos et al. BID9 suggest based on extensive experimental evaluation that good generalization ability is linked to reduced selectivity of learned features. They further suggest that individual selective units do not play a strong role in the overall performance on the task as compared to the less selective ones. They connect the ablation of selective features to the heuristics employed in neural network feature pruning literature which prune features whose removal does not impact the overall accuracy significantly BID8 BID2. The findings of Zhou et al. BID13 concur regarding the link between emergence of feature selectivity and poor generalization performance. They further show that ablation of class specific features does not influence the overall accuracy significantly, however the specific class may suffer significantly. We show that the emergence of selective features in Adam, and the increased propensity for pruning the said selective features when using L2 regularization may thus be helpful both for better generalization performance and network speedup. Our findings would help practitioners and theoreticians be aware that seemingly unrelated hyperparameters can inadvertently affect the underlying network capacity, which interplays with both the test accuracy and generalization gap, and could partially explain the practical performance gap between Adam and SGD. Our work opens up future avenues of theoretical and practical exploration to further validate our hypotheses, and attempt to understand the emergence of feature selectivity in Adam and other adaptive SGD methods. As for network speed up due to sparsification, the penalization of selective features can be seen as a greedy local search heuristic for filter pruning. While the extent of implicit filter sparsity is significant, it obviously does not match up with some of the more recent explicit sparsification approaches BID1 BID3 which utilize more expensive model search and advanced heuristics such as filter redundancy. Future work should reconsider the selective-feature pruning criteria itself, and examine nonselective features as well, which putatively have comparably low discriminative information as selective features and could also be pruned. These non-selective features are however not captured by greedy local search heuristics because pruning them can have a significant impact on the accuracy. Though the accuracy can presumably can be recouped after fine-tuning.

Represent the TLDR for finding the corresponding scientific article:

Filter level sparsity emerges implicitly in CNNs trained with adaptive gradient descent approaches due to various phenomena, and the extent of sparsity can be inadvertently affected by different seemingly unrelated hyperparameters.

Represent the short summary for finding the corresponding scientific article (long):


Question: Represent text.

Question: Humans can learn a variety of concepts and skills incrementally over the course of their lives while exhibiting an array of desirable properties, such as non-forgetting, concept rehearsal, forward transfer and backward transfer of knowledge, few-shot learning, and selective forgetting. Previous approaches to lifelong machine learning can only demonstrate subsets of these properties, often by combining multiple complex mechanisms. In this Perspective, we propose a powerful unified framework that can demonstrate all of the properties by utilizing a small number of weight consolidation parameters in deep neural networks. In addition, we are able to draw many parallels between the behaviours and mechanisms of our proposed framework and those surrounding human learning, such as memory loss or sleep deprivation. This Perspective serves as a conduit for two-way inspiration to further understand lifelong learning in machines and humans. Humans have a sustained ability to acquire knowledge and skills, refine them on the basis of novel experiences, and transfer them across domains over a lifespan. It is no surprise that the learning abilities of humans have been inspiring machine learning approaches for decades. Further extending the influence of human learning on machine learning, continual or lifelong learning (LLL) in particular, is the goal of this work. To mimic human continual concept learning scenarios, in this paper we propose a new learning setting in which one concept-learning task is presented at a time. Each classification task and its corresponding labeled data are presented one at a time in sequence. As an example, assume we are given the task sequence of learning to classify hand-written characters of "1", "2", "3", and so on, by providing training examples of "1", "2", "3" in sequence (such as those in Figure 1). When learning "1", only data of "1" is available. When learning of "2", data of "2" becomes available, and so on. Preprint. Under review. We assume that these concepts are mutually exclusive (i.e. any sample can be a positive sample for at most one task). We also assume a set of " negative" examples are given -they are presumably the previously learned concepts. This setting is in stark contrast to the standard setup in multi-class machine learning, where it is assumed that all training data of all classes is readily available, as a "batch" mode. For the example task sequence in this section, we can consider the four classes to be "1", "2", "3", and "I". The negative samples used when learning to identify the three positive classes can come from a domain-appropriate non-overlapping "" set (in this case, lower-case letters). After a model is trained up to class i, it will not have access to those training samples unless absolutely necessary (see later). The samples shown are from the EMNIST dataset. Under this continual learning setting, we hope than a LLL approach would exhibit the following properties: Non-forgetting: This is the ability to avoid catastrophic forgetting of old tasks upon learning new tasks. For example, when learning the second task "2", with only training data on "2" (and without using the data of the task 1), "2" should be learned without forgetting how to classify task 1 learned earlier. Due to the tendency towards catastrophic forgetting, non-lifelong learning approaches would require retraining on data for "1" and "2" together to avoid forgetting. A skill opposite to non-forgetting is selective forgetting. As we will describe further, learning new tasks may require expansion of the neural network, and when this is not possible, the model can perform selective forgetting to free up capacity for new tasks. Forward transfer: This is the ability to learn new tasks easier and better following earlier learned tasks. For example, after learning the task of classifying "1", it would be easier (requiring less training data for the same or higher predictive accuracy) to learn to classify "I". Achieving sufficient forward transfer opens the door to few-shot learning of later tasks. Non-confusion: Machine learning algorithms need to find discriminating features for classification only as robust as they need to be to minimize a loss, thus, when more tasks emerge for learning, earlier learned features may not be sufficient, leading to confusion between classes. For example, after learning "1" and "2" as the first two tasks, the learned model may say "with only straight stroke" is "1" and "with curved stroke" is "2". But when learning "I" as a later new task, the model may rely only on the presence of a straight stroke again, leading to confusion between "1" and "I" when the model is finally tested. To resolve such confusion between "1" and "I", samples of both "1" and "I" are needed to be seen together during training so that discriminating features may be found. In humans, this type of confusion may be seen when we start learning to recognize animals for example. To distinguish between common distinct animals such as birds and dogs, only features such as size or presence of wings is sufficient, ignoring finer features such as facial shape. However, when we next learn to identify cats, we must use the previous data on dogs and new data on cats to identify finer features (such as facial shape) to distinguish them. Backward transfer: This is knowledge transfer in the opposite direction as forward transfer. When new tasks are learned, they may, in turn, help to improve the performance of old tasks. This is analogous to an "overall review" before a final exam, after materials of all chapters have been taught and learned. Later materials can often help better understand earlier materials. Past works on LLL have only focused on subsets of the aforementioned properties. For example, an approach inspiring our own, Elastic Weight Consolidation (EWC), focuses only on non- forgetting. The approach of considers non-forgetting as well as forward and backward transfer and confusion reduction, but does not allow for selective forgetting. Figure 2 illustrates the scope of our framework compared with related previous approaches. Section 4 contains a more detailed comparison. In this paper, we provide a general framework of LLL in deep neural networks where all of these these abilities can be demonstrated. Deep neural networks, which have become popular in recent years, are an attractive type of machine learning model due to their ability to automatically learn abstract features from data. Weights (strengths of links between neurons) of a network can be modified by the back propagation algorithms to minimize the total error of the desired output and the actual output in the output layer. In our study, we consider fully-connected neural networks with two hidden layers to illustrate our LLL approach. The basic idea of our unified framework, similar to EWC, is to utilize "stiffness" parameters of weights during training phases to achieve the various LLL properties such as nonforgetting, forward transfer, etc. For each lifelong learning property, a subset of its weights may be "frozen", another subset of weights may be "free" to be changed, and yet another subset of weights may be "easily changed", depending on the type of lifelong learning properties we are aiming to facilitate at the time. EWC and its conceptual successors are lifelong learning approaches which estimate the importance of each weight in a network for maintaining task performance. By preventing already important weights from changing to accommodate new tasks (i.e. consolidating weights), catastrophic forgetting can be reduced. Generally speaking, each network weight, θ i, is associated with a remembering, BWT+: positive backward transfer, FWT: forward transfer. We expect our approach to be outperform the related approaches of EWC and PNN on a majority of the metrics. consolidation value, b i, which can be set or tuned for each stage of learning as we will soon discuss. When training a model with EWC, we combine the original loss L t with weight consolidation as follows: Here, θ target i is the consolidation target value for a weight, θ t i is the weight value being updated during training on task t, and λ is used to balance the importance of the two loss components. Clearly, a large b value indicates that changing the weight is strongly penalized during training, whereas a value of 0 indicates that the weight is free to change. In our approach, we use three values for b to control the flexibility of different sets of network weights: • b nf for non-forgetting (ideally a very large value), • b tr for forward transfer (ideally very small or zero), • and b f ree for freely tunable weights (ideally very small). While the individual weights of the network are learned via back-propagation, these consolidation hyperparameters are set by several heuristic strategies. We will illustrate how changing these hyperparameters to control the stiffness of weights in different parts of the deep neural networks, our approach can achieve all of the LLL abilities mentioned above (Section 2). As we mentioned, these hyperparameters are determined during LLL by heuristic strategies, but one might wonder if these heuristics can be learned. Our comparison between lifelong learning of machines and humans suggests that our model hyperparameters are probably intrinsic to the physiology of the brain -a product of natural evolution. A person can consciously perform metalearning, such as during memory training and explicit rehearsal, whereas these heuristics may be explicitly learned or fine-turned. This would be our future study. To provide an intuitive understanding of our approach, we will describe how it would be applied to learning a sequence of hand-written numbers (such as "1", "2", "3") The training data at each step consists of positive examples for the class as well as samples from a pool of negative examples, such as hand-written small letters. These classes are different from the new classes to be learned and may be previously learned by the network. Figure 1 ) displays samples from these discussed classes. For a neural network architecture, we will consider for illustrative purposes a simple feed-forward network with two fully-connected hidden layers trained with gradient descent. This approach conceptually extends to more complex architectures including Convolutional Neural Networks (CNNs), whereby it can be applied to improving the training and inference costs associated with many tasks that traditionally require large networks and long training times, common in computer vision tasks. As mentioned earlier, several heuristic strategies are used to set the hyperparameters in the deep neural network so that it can exhibit various LLL properties. However, the purpose of this section is not about those heuristics -they will be described in detail with experimental in a followup technical paper. Instead, we will describe the setting (or sizes) of those hyperparameters to illustrate their ability to demonstrate the desired LLL properties. Nevertheless, we list those heuristics briefly below: • When a new task is being trained with data, its associated weight consolidation b values would be set to 0 or small to allow easy learning of the task; • After a new task is learned, its associated b values should be set to be very large so that it will not to be "forgotten" or affected when other tasks are being learned; • If a new task is similar (by some task similarity measure) to some old tasks, the b values of the forward transfer links from the old tasks to the new one should be set to 0 or very small, to encourage forward transfer of the representation and knowledge, realizing fewshot learning of the new task; • If two tasks are confused, the b values of their associated weights should be set to be small so that confusion can be resolved by rehearsing on their training data. The b values of other tasks will be set to be very large so that other tasks will not be affected. • If a certain task is not often used (which can be measured by the number of times it is tested or used in deployment), its associated b values will be gradually reduced. This amounts to the controlled forgetting of the task while other new tasks are being learned. This is useful when the network reaches its size or computational capacity when it learns new tasks. In Section 3 we will draw parallel in the setting of these hyperparameters in our model with human learning phenomena, such as memory loss and sleep deprivation. We assume that we start with a small or medium sized network for task 1. As discussed in the next step, this network size does not need to be fixed, and will increase as more tasks are learned. Alternatively, we can start with a very large network, and after learning each task, prune the network while maintaining performance. For this simplified example, will use network expansion to illustrate our framework. Training the network for class 1 is performed in the standard way: all weights of the neural network are free to modify. In Figure 3a, this is reflected as all the weights connecting groups of neurons in sequential layers being blue. Training data consists of class 1 and samples. In preparation for training class 2, to achieve non-forgetting of class 1, we increase the network capacity by introducing a set of free neurons for task 2. The number of new neurons is determined by the difficulty of task 2 as well as how similar task 2 is to the previous tasks. The more difficulty the task 2, or the more dissimilar it is from the previous tasks, the more new neurons. This will be further discussed in our forthcoming paper. We set the consolidation strength of all weights used for task 1 to be very large (as b nf), while for task 2 to be small. This would translate into little influence on class 1's performance when training with back propagation on task 2, without using the task 1 data. This achieves non-forgetting for task 1. The strongly consolidated weights are reflected in Figure 3b with the red weights. The b values of weights between these newly added nodes and the input and output layers are set to be small so weights can updated via back-propagation (blue in Figure 3b). To encourage the forward transfer of skills from task 1 to task 2 if they are similar, we set the b of weights pointing from old nodes to new nodes to be very small (or 0), as b tr (green). By allowing forward transfer to occur from every previous task to the newest task, few-shot learning can occur to increasing extents, allowing new tasks to be learned with less data than previous tasks. Through leveraging curriculum learning at both the level of task samples and perhaps individual tasks, the degree of forward transfer and few-shot learning capacity may be further improved. During training of class 2, only the loss from the class 2 output needs to be back-propagated. As task 1 has been given and learned, we can mix in a small amount of class 1 samples with the samples to be negative examples of class 2, to allow better discrimination of class 1 and 2. This would reduce the amount of confusions between tasks 1 and 2 that need to be resolved (see Section 2.4). To prepare for class 3 training, again apply a consolidation strength of b nf to all weights used by previous classes and extend each layer with new nodes for learning class 3. This time, forward transfer connections are added from both class 1 and class 2 node groups to the new class 3 nodes, as pictured in Figure 3c. Again, during training of class 3, only the loss from the class 3 output needs to be back-propagated. By this point, the network may have learned how to minimally discriminate "1" and "2" by identifying if there is only straight stoke or not. As per the previous step, tasks 3 has also been individually learned well. If all classes were visually distinct, the network may perform well when being tested on all classes together, however, if we assume that task 3 is "I", making it visually similar to class 1, confusion may occur between these two classes. That is, when "I" is presented in the input, both the output for "1" and output for "I" will be activated, indicating a classification of "1" or class "I", and thus, the prediction can be incorrect or unreliable. This is when confusion has happened. In this case, the data for class "1" and class "I" will be used to train the network to resolve confusion. We will use the network with its consolidation values as 7 in Figure 4a. If the existing network weights are insufficient to reduce the confusion, then additional weights can be added to the network. Note that back-propagation of errors only need to be applied to outputs for the confused classes (classes "1" and "I" in this case). As consolidation of weights important to class "2" is set to be very large with b nf, its performance should not be significantly affected while confusion is resolved between "1" and "I". In practice, when individual tasks are being learned in sequence as described earlier, confusion can happen between any pair of tasks. To handle this, pair-wise confusion reduction as described here can be applied to resolve all pair-wise confusions, starting with the pairs with the highest confusion, and continuing until some stopping criterion is reached. Note that weights important to preserving performance on class 2 are frozen with b nf. To provide any additional capacity needed for resolving confusion, the column of yellow nodes can be added (and later grouped with the class 1 node column). In (b): consolidation of weights during backward transfer for overall refinement. Note the addition of backward transfer weights from 3 to 1&2, and 2 to 1. All weights are free to change while rehearsal is done on a fraction of training samples for all classes seen so far. This step facilitates further non-forgetting, forward transfer, confusion reduction, as well as backward transfer. In (c), selective forgetting of task 1 is performed so that an additional task 4 can be learned without the addition of many new weights. This will cause forgetting of task 1 gradually and may slightly affect those tasks relying on forward transfer from task 1 skills. By this point in training: skills learned for earlier classes have been utilized by later classes (ex. 2 benefits from 1, 3 benefits from 1 and 2), but later skills have not been able to transfer to earlier classes. To achieve such backward transfer of skills as well as overall refinement of the model, we can first initialize the backward transfer weights pointing from nodes originally added for each class to those 8 nodes for classes before it (i.e. from 3 to 1 and 2, from 2 to 1), and give them all a consolidation value of b f ree as demonstrated in Figure 4b. Tuning is then performed on a small fraction of training samples for each class at the same time. A smaller b f ree value at this stage is expected to allow refinement to occur more easily. Note that even though this backward transfer looks similar to batch learning of all tasks, it is a final fine-tuning process which does not usually incur a significant computational cost, as the network has been "pre-trained" on each task individually. If we have reached a size limit for the network, but still want to learn additional tasks, we can perform selective forgetting to free up network capacity from previous tasks. To perform selective forgetting, we determine which tasks are the least important. This can be done using a heuristic method such as keeping track of usage frequency. Forgetting unused tasks would be similar to how we may forget a face or name if the associated person has not been seen for a long time. In our running example, let us assume the least important task is task 1. Before we start training on a new task (task 4), we again apply b nf to weights used by previous classes, however, for those nodes used primarily by task 1, the weights between them are unfrozen with a consolidation of b f ree. Doing so sacrifices performance on task 1 and possibly a small amount of performance on tasks obtaining forward transference from task 1 in exchange for the ability to continue learning new tasks. The consolidation settings of weights when learning some further task 4 can be seen in Figure 4c. Note that when the network capacity is reached but we wish to learn additional tasks, network capacity for previous tasks must be sacrificed for new tasks. With b values set to be small for task 1, task 1 is naturally and gradually forgotten when the new tasks are learned. There is not necessarily a clear point at which task 1 is forgotten. This is also similar to human memory when one can only recall the face of an old friend vaguely. Assume that there are a total of T tasks. Training T tasks individually and incrementally (as has been described earlier in this section) needs O(T) time complexity in total. Additional computation is needed for confusion reduction and backward transfer. However, if we assume tasks are highly distinctive, then confusion would happen infrequently. The final backward transfer can resolve any leftover errors in all tasks. As the network has been pre-trained by all tasks, hopefully the final backward transfer would not require too much computation. On the other hand, with proper forward transfer and backward transfer of knowledge, the overall training data needed would be smaller, thus requiring less learning time. With regard to the suite of LLL metrics proposed by and displayed in Figure 2, we expect that by intelligently encouraging forward and backward transfer and reducing rehearsal and sample storage to minimally necessary levels, we can outperform related work in a majority of metrics. Detailed empirical and theoretical analysis will however be left to our future manuscript. Resourceful and versatile large small yes large Alzheimer's disease small --large Table 1: Correspondences between settings that can be used in our approach and behavioural descriptions. For example, being able to transfer knowledge across tasks and from old tasks to a newly encountered one allows us to be versatile. This ability to transfer knowledge is analogous to a small b tr value in our approach. As our approach works at the level of controlling the flexibility of individual network weights, it is natural to ask how our approach lines up with the mammalian brain whose lifelong learning behaviours we are attempting to capture. In this section we consider the parallels between our model and human learning with respect to several behavioural patterns listed in Table 1. For each of these, we will discuss how a similar behaviour can be exhibited by our model with specific hyperparameter settings, and how it may translate to what happens in the human brain. Resourceful and versatile. Ideally, human learning allows us to acquire a lot of knowledge, yet be flexible enough to adapt to new experiences and make meaningful connections between experiences. The ability to make connections between different events and skills is roughly analogous to forward and backward transference explicitly considered by our approach, which is modulated though the b tr hyperparameter and rehearsal. Being able to remember important information for long spans of time corresponds to a large b nf value, while also being able to adapt to new tasks is facilitated by adding new nodes to the network for each task. Memory loss. In our approach, a large b nf consolidation value is applied to weights when memories need to remain unchanged. Thus, when the value for b nf is not chosen properly, the model will lack the ability to remember tasks. A small or medium b nf value, where memories are easily replaced with new ones, may appear similar to gradual memory loss in humans, with smaller b nf values corresponding to faster loss. Sleep deprived. The human brain is suspected of performing important memory-related processes during sleep, and sleep deprivation has been observed to be detrimental to cognitive performance related to memory. Confusion reduction and backward transfer are important stages of our proposed approach which utilize rehearsal (functionally similar to memory replay) -where the model is exposed to samples from past tasks in order to perform fine-tuning to achieve various properties. Without these rehearsal-utilizing steps, the model may be less able to distinguish between samples of similar classes. Additionally, the ability to identify connections between newer tasks and older ones will be lost, so that potentially useful newly acquired skills cannot benefit older tasks. "Rain man". By setting b tr to be a large value and not performing the forward and backward transfer step, skill transfer between tasks will not occur. If b nf is still large, this leads to a model which is very good at rote learning -remembering individual tasks, but unable to generalize knowledge to new or old tasks. This is reminiscent of Kim Peek, who was able to remember vast amounts of information, but performed poorly at abstraction-related tasks. Alzheimer's disease. Usually an early stage of Alzheimer's disease is characterized by good memory on events years ago but poor on recent events. This can be modeled in our framework by a small number of new neurons for new tasks and a very large b nf for old ones. Much recent work on LLL has been done, with a thorough review of this work available in. However, it seems that no existing approaches simultaneously address all of the LLL abilities of our proposed approach. LLL approaches can be grouped by their primary mechanism. Namely, we consider the approach groups most relevant to the present work: parameter-based, rehearsal-based, and dynamic network-based. Parameter-based approaches. Parameter-based LLL approaches aim to identify the important weights of a neural network and prevent their modification in order to avoid catastrophic forget- ting. An early influential approach of this type is Elastic Weight Consolidation, which uses Fisher information to estimate weight importance and employs a quadratic loss to penalize weight changes during training. Aiming to bring some of the biological complexity of real synapses to neural networks, proposes an approach which allows each weight to estimate its own importance during the training process. Theoretical improvements and generalization of can be found in. Uniquely, presents an unsupervised approach to calculate weight importance, which works by estimating the sensitivity of the learned function with respect to each weight (as opposed to the sensitivity with regard to the loss function, as in ). Orthogonal Weights Modification can be viewed as a type of parameter-based approach. While the technique does not entirely prevent the modification of important weights, it aims to restrict their movement to directions where forgetting will not occur. Similar to existing parameter-based approaches, the present approach uses regularization to prevent the modification of important weights, and similar to, we design our approach to not only combat catastrophic forgetting, but also reduce intransigence and confusion. However, unlike existing approaches, we leverage regularization for weight consolidation in a more flexible manner by modulating the consolidation strengths of subsets of weights as necessary for maintaining some task skills while allowing other tasks to benefit from transference and refinement. Similar to, we also incorporate the ability to selectively forget tasks, but implement it in a more controlled fashion so that forgetting of memories can be done according to any desired policy (not just slowly fading as task skills go unused). Rehearsal-based approaches. Rehearsal is one of the earliest strategies employed to combat catastrophic forgetting. These approaches generally work by storing some amount of past task data to be incorporated into further training, so that information about old task data distributions is never fully lost (as can happen in parameter-based approaches). When learning new tasks, many rehearsalutilizing approaches require samples from all previous tasks. For example, Gradient Episodic Memory (GEM) and its more efficient successor Averaged GEM require samples from all previous tasks in order to to restrict the new-task loss so that the loss on old tasks does not increase. Meta-Experience Replay (MER) similarly requires old task samples to perform gradient alignment and to mix in with new samples during training. Old task samples are also required by iCaRL to compute a distillation loss during new task training. In contrast, our unified approach does not require old task data while learning new tasks, only using this data sparingly to reduce pairwise class confusions after the new task learning, and when performing backward transfer. Similar to our approach, does not necessarily require old samples when learning a new task. This is done through the observation that the last fully-connected layer of a lifelong learning neural network has a bias towards newer classes. The authors correct this bias with a two-parameter linear model per task, which is tuned using both previous and new task samples. Dynamic network-based approaches. While consolidation-based and rehearsal-based LLL approaches aim to condense information about several tasks into the same set of weights, dynamic network-based approaches generally take the route of extending the network for new tasks. Progressive Neural Networks (PNNs) are a prime example of this type of approach. Similar to our approach, PNNs start out with a single column (one set of neural network layers) to use for learning the first task, and for each subsequent task, a new column is added while all previous column weights are frozen during training (making PNNs immune to forgetting, but not necessarily confusion). Lateral connections from every past column to the newest column are also added to facilitate forward transfer. However, unlike our approach, we add lateral connections from new columns to older ones to facilitate backward transfer and propose a method to solve the confusion that comes with single-head evaluation (while considers only multi-head evaluation). A primary criticism of PNNs is the quickly growing network size that comes with added a fixed, possibly redundant set of nodes for each new task. By supporting selective forgetting and variable-size additional columns, our proposed approach partially deals with this increasing network size by allowing parameters of less important tasks to effectively be recycled, and by adding less capacity for easier tasks. Improving upon PNNs, several approaches with more efficient resource allocation have been proposed. Dynamically Expandable Networks (DENs) extend each layer of the network only as much as necessary, and abstractly works by identifying which existing parameters can be used as-is, and when a parameter would be substantially changed to adapt to the new task, a duplicate of the associated neuron is retrained so that the network has a version of the neurons tuned for previous tasks, and a version tuned for the new task. After training on each task, fine-tuning on all tasks is performed, making DEN also fall under the rehearsal-utilizing group of approaches. Naturally, DEN is effective at facilitating forward transfer, however, it does not consider backward transfer and modifying the approach for selective forgetting may pose a challenge. Reinforced Continual Learning (RCL) aims to optimize the expansion amount of each layer to accommodate each new task using reinforcement learning, and aims to balance accuracy on new task with added network complexity. Unlike DEN, RCL keeps the learned parameters for previous tasks fixed and only updates the added parameters. Similar to both DEN and PNNs, RCL does not account for either backward transfer, confusion reduction, or selective forgetting. In this work, we presented a unified approach for lifelong learning. This approach tackles a difficult problem that captures many important aspects of human learning, namely non-forgetting, forward transfer, confusion reduction, backward transfer, few-shot learning, and selective forgetting. Progress in this area is critical for the development of computationally efficient and flexible machine learning algorithms. The success at this problem reduces the demand for training data while a single model learns to solve more and more tasks. While previous works have focused on a subset of these lifelong learning skills, our proposed approach utilizes a single mechanism, controlling weight consolidation, to address all of the considered skills. We define only a small number of consolidation hyperparameters which are dynamically applied to groups of weights. In addition to describing the novel approach, we examine its parallels with human learning. We note several similarities in the response of our model to hyperparameter settings and the effects on human learning to analogous changes in the brain.

Represent the short summary for finding the corresponding scientific article (long)

Drawing parallels with human learning, we propose a unified framework to exhibit many lifelong learning abilities in neural networks by utilizing a small number of weight consolidation parameters.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Filter level sparsity emerges implicitly in CNNs trained with adaptive gradient descent approaches due to various phenomena, and the extent of sparsity can be inadvertently affected by different seemingly unrelated hyperparameters.


Question:
represent the scientific article to find a short tldr

E.g.
'We review three limitations of BLEU and ROUGE – the most popular metrics used to assess reference summaries against hypothesis summaries, come up with criteria for what a good metric should behave like and propose concrete ways to assess the performance of a metric in detail and show the potential of Transformers-based Language Models to assess reference summaries against hypothesis summaries. Evaluation metrics play a central role in the machine learning community. They direct the efforts of the research community and are used to define the state of the art models. In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU [] and ROUGE []. Both approaches rely on counting the matching n-grams in the candidates summary to n-grams in the reference text. BLEU is precision focused while ROUGE is recall focused. These metrics have posed serious limitations and have already been criticized by the academic community [] [] [] []. In this work, we formulate an empirical criticism of BLEU and ROUGE, establish a criteria that a sound evaluation metric should have and propose concrete ways to test any metric towards these criteria. We also use recent advances in NLP to design a data-driven metric addressing the weaknesses found in BLEU and ROUGE and scoring high on the criteria for a sound evaluation metric. 2 Related Work 2.1 BLEU, ROUGE and n-gram matching approaches BLEU (Bilingual Evaluation Understudy) [] and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [] have been used to evaluate many NLP tasks for almost two decades. The general acceptance of these methods depend on many factors including their simplicity and the intuitive interpretability. Yet the main factor is the claim that they highly correlate with human judgement []. This has been criticised extensively by the literature and the shortcomings of these methods have been widely studied. Reiter [], in his structured review of BLEU, finds a low correlation between BLEU and human judgment. Callison et al [] examines BLEU in the context of machine translation and find that BLEU does neither correlate with human judgment on adequacy(whether the hypothesis sentence adequately captures the meaning of the reference sentence) nor fluency(the quality of language in a sentence). Sulem et al [] examines BLEU in the context of text simplification on grammaticality, meaning preservation and simplicity and report BLEU has very low or in some cases negative correlation with human judgment. Language modeling has become an important NLP technique thanks to the ability to apply it to various NLP tasks as explained in Radford et al []. There are two leading architectures for language modeling Recurrent Neural Networks (RNNs) [] and Transformers []. RNNs handle the input tokens, words or characters, one by one through time to learn the relationship between them, whereas, transformers receive a segment of tokens and learn the dependencies between them using an attention mechanism. While BLEU and ROUGE are defined in a discrete space new evaluation metric can be defined in this continuous space. BERTscore [] uses word embeddings and cosine similarity to create a score array and use greedy matching to maximize the similarity score. Sentence Mover's Similarity [] uses the mover similarity, Wasserstein distance, between sentence embedding generated from averaging the word embeddings in a sentence. One other evaluation method proposed is RUSE [] this method proposes embedding both sentences separately and pooling them to a given size. After that they use a pre trained MLP to predict on different tasks. This quality estimator metric is then proposed to be used in language evaluation. Our proposed methodology is to take neural language evaluation beyond architecture specifications. We are proposing a framework in which an evaluator's success can be determined. In this part, we discuss three significant limitations of BLEU and ROUGE. These metrics can assign: High scores to semantically opposite translations/summaries, Low scores to semantically related translations/summaries and High scores to unintelligible translations/summaries. Suppose that we have a reference summary s1. By adding a few negation terms to s1, one can create a summary s2 which is semantically opposite to s1 but yet has a high BLEU/ROUGE score. In addition not to be sensitive to negation, BLEU and ROUGE score can give low scores to sentences with equivalent meaning. If s2 is a paraphrase of s1, the meaning will be the same;however, the overlap between words in s1 and s2 will not necessarily be significant. A third weakness of BLEU and ROUGE is that in their simplest implementations, they are insensitive to word permutation and can give very high scores to unintelligible sentences. Although higher order BLEU scores are expected to mitigate this effect, they make the metric more sensitive to paraphrasing. To overcome the previously highlighted challenges and provide a framework by which metrics comparing reference summaries/translation can be assessed and improved, we established firstprinciples criteria on what a good evaluator should do. The first one is that it should be highly correlated with human judgement in semantic similarity. The second one is that it should be able to distinguish sentences which are in logical contradiction, logically unrelated or in logical agreement. The third one is that given s1, s2 which are semantically similar, eval(s1,s2) > eval(s1,s2(corrupted) > eval(s1,s2(more corrupted)) where corruption here includes removing words, adding noise to the word order or including grammatical mistakes. We will now give a more detailed example to how the scorecard can be implemented. For every dimension of the scorecard the experiments are done with three metrics. BLEU with equal weights between 1 to 4 grams. ROUGE with averaging ROUGE-1 and ROUGE-2 and the a neural evaluator. The evaluator is the RoBERTa large pre-trained model, which we fine tune it to predict sentence similarity (0-5 scale) on the STS-B benchmark dataset (8628 sentence pairs). The first expectation from a google similarty metric is to correlate highly with human judgment in terms of assessing semantic similarity. Here we assessed BLEU and ROUGE on the STS-B benchmark and compared their performance to a RoBERTa model fine tuned for semantic similarity (Table 1). Another characteristic of a good metric is to differentiate the argument, core meaning, in a sentence and take it into account when assessing hypothesis text with references. Here we used the MNLI dataset where for each text we have three hypothesis text representing contradiction, neutral and entailment. We expect a good metric to rank entailment higher than neutral and both of them higher than contradiction. To assess the quality of a metric we propose to use the Spearman's ranked correlation and in Table 4.2.2 we also experiment with Kendall's τ. Here we observe that the RoBERTa model remarkably outperforms BLEU and ROUGE and both of these metrics show very little correlation with human judgment. For assessing the third criteria. We start with 3479 sentence pairs from the MNLI dataset that are labelled as entailment. We introduce random corruptions such as random insertion, deletion and grammatical errors as in []. We use two different set of parameters for different corruption levels and expect that a good metric would rank the original similar sentence higher than the less corrupted and both higher than the more corrupted sentence. Here we also propose to use the Spearman's ranked correlation and also experiment with Kendall's τ. We report on In this work, we have established a framework to assess metrics comparing the quality of reference and hypothesis summary/translations. Based on these criteria, we compare evaluators using recent Transformers to BLEU and ROUGE and highlight their potential to replace BLEU and ROUGE.' == 'New method for assessing the quaility of similarity evaluators and showing potential of Transformer-based language models in replacing BLEU and ROUGE.' != 'This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network. '

Question: Limited angle CT reconstruction is an under-determined linear inverse problem that requires appropriate regularization techniques to be solved. In this work we study how pre-trained generative adversarial networks (GANs) can be used to clean noisy, highly artifact laden reconstructions from conventional techniques, by effectively projecting onto the inferred image manifold. In particular, we use a robust version of the popularly used GAN prior for inverse problems, based on a recent technique called corruption mimicking, that significantly improves the reconstruction quality. The proposed approach operates in the image space directly, as a of which it does not need to be trained or require access to the measurement model, is scanner agnostic, and can work over a wide range of sensing scenarios. Computed Tomography (CT) reconstruction is the process of recovering the structure and density of objects from a series of x-ray projections, called sinograms. While traditional full-view CT is relatively easier to solve, the problem becomes under-determined in two crucial scenarios often encountered in practice -(a) few-view: when the number of available x-ray projections is very small, and (b) limited-angle: when the total angular range is less than 180 degrees, as a of which most of the object of interest is invisible to the scanner. These scenarios arise in applications which require the control of x-ray dosage to human subjects, limiting the cost by using fewer sensors, or handling structural limitations that restrict how an object can be scanned. When such constraints are not extreme, suitable regularization schemes can help produce artifact-free reconstructions. While the design of such regularization schemes are typically driven by priors from the application domain, they are found to be insufficient in practice under both few-view and limited-angle settings. In the recent years, there is a surge in research interest to utilize deep learning approaches for challenging inverse problems, including CT reconstruction. These networks implicitly learn to model the manifold of CT images, hence ing in higher fidelity reconstruction, when compared to traditional methods such as Filtered Backprojection (FBP), or Regularized Least Squares (RLS), for the same number of measurements. While these continue to open new opportunities in CT reconstruction, they rely of directly inferring mappings between sinograms and the corresponding CT images, in lieu of regularized optimization strategies. However, the statistics of sinogram data can vary significantly across different scanner types, thus rendering reconstruction networks trained on one scanner ineffective for others. Furthermore, in practice, the access to the sinogram data for a scanner could be restricted in the first place. This naturally calls for entirely image-domain methods that do not require access to the underlying measurements. In this work, we focus on the limited-angle scenario, which is known to be very challenging due to missing information. Instead of requiring sinograms or scanner-specific representations, we pursue an alternate solution that is able to directly work in the image domain, with no pairwise (sinogram-image) training necessary. To this end, we advocate the use of generative adversarial networks (GANs) as image manifold priors. GANs have emerged as a powerful, unsupervised technique to parameterize high dimensional image distributions, allowing us to sample from these spaces to produce very realistic looking images. We train the GAN to capture the space of all possible reconstructions using a training set of clean CT images. Next, we use an initial seed reconstruction using an existing technique such as Filtered Back Projection (FBP) or Regularized Least Squares (RLS) and'clean' it by projecting it onto the image manifold, which we refer to as the GAN prior following. Since the final reconstruction is always forced to be from the manifold, it is expected to be artifact-free. More specifically, this process involves sampling from the latent space of the GAN, in order to find an image that resembles the seed image. Though this has been conventionally carried out using projected gradient descent (PGD), as we demonstrate in our , this approach performs poorly when the initial estimate is too noisy or has too many artifacts, which is common under extremely limited angle scenarios. Instead, our approach utilizes a recently proposed technique referred to as corruption mimicking, used in the design of MimicGAN, that achieves robustness to the noisy seed reconstruction through the use of a randomly initialized shallow convolutional neural network (CNN), in addition to PGD. By modeling the initial guess of this network as a random corruption for the unknown clean image, the process of corruption mimicking alternates between estimating the unknown corruption and finding the clean solution, and this alternating optimization is repeated until convergence, in terms of effectively matching the observed noisy data. The ing algorithm is test time only, and can operate in an artifact-agnostic manner, i.e. it can clean images that arise from a large class of distortions like those obtained from various limited-angle reconstructions. Furthermore, it reduces to the well-known PGD style of projection, when the CNN is replaced by an identity function. We restrict our study to parallel beam and fan beam types of scanners, that produces a CT reconstruction in a 2D slice-by-slice manner. The CT reconstruction problem, like most other inverse problems, can be written as: X * = arg min X A(X) − y + R(X), where X ∈ R d×d is the image to be reconstructed, y ∈ R v×d is the projection, referred to as a "sinogram", and A is the x-ray projection operator of the particular CT scanner. Here, the number of available x-ray projections is given by v, and the number of detector columns is given by d. Note, A(X) can be written as a matrix multiplication, but the matrix tends to be a sparse, very large matrix. Here, for simplicity we denote it as an operator acting on X. Typically, a regularization function in the form of R(X) is used to further reduce the space of possible solutions. In order to get a complete faithful reconstruction of X, the object must be scanned from a full 180 •. When the viewing angle is much lesser than << 180 •, most existing methods return an X * that is extremely corrupted by noise and missing edges, with little or no information of the original structure present. While several kinds of regularization functions have been used (for e.g. total variation and its variants), in this paper we advocate the use of R(X) such that it forces X to be from a known image manifold. We achieve this by using generative adversarial networks (GANs), which have emerged as a powerful way to represent image manifolds. In particular at test time, given a sinogram y, the problem can be formulated as A(G(z)) − y, where G is a pre-trained generator, and finally, X * = G(z *) and can be solved using stochastic gradient descent. This has been referred to as a GAN prior or a manifold prior for inverse imaging. However, solving the equation of the form in is not always possible since one may not have access to the measurement model A. A more accessible (yet different) form that does not require A to be known is given by: In this work we obtain the initial estimate X RLS using regularized least squares (RLS) approach. As a of, the quality of the final estimate largely depends on the quality of the initial reconstruction. Particularly, if the estimate is very noisy or poor, as is the case for limited angle CT, the optimization in can easily fail, especially when the loss is not robust to the type of corruption noise or distortion. In scenarios of interest in this paper, even a powerful regularizer such as the GAN prior can fail due to a poor initial estimate. In order to avoid this, we propose to use a recently proposed modification of the GAN prior, that performs better even with heavily distorted images. The process called corruption mimicking was proposed in, was designed to improve the quality of projection onto the manifold under a variety of corruptions. Corruption Mimicking and the Robust GAN Prior: Let us suppose X RLS = f (X *), where f is an unknown distortion or corruption function, and X * is the unknown global optima to. Corruption Mimicking is the process of estimating both X * and f simultaneously, using a shallow neural network to approximate f with a few examples. As a , we now modify as follows: Equation is solved using alternating optimization, where we first solve for the optimalf * conditioned on the current estimate X *, and repeat the process until convergence. Since we constrain f to be shallow, even as few as 100 samples are sufficient. In our setting,f contains 2 convolutional layers with ReLU activations, followed by a masking layer (pixel-wise multiplication). Finally, we also include a shortcut connection at the end to encourage it to learn identity. The GAN prior now becomes a special case of the Robust GAN prior, whenf = I, the identity function. An appealing property of this technique is that it is corruption-agnostic i.e., the same system can be reused to obtain accurate CT reconstructions across a wide variety of limited-angle settings. We test the effectiveness of the robust GAN prior by performing CT reconstruction of the MNIST and Fashion-MNIST datasets. We first project these datasets into their projection space (sinograms), using a forward projection operation, to simulate the CT-scan process. While we consider a parallel beam scanner in these experiments, the methods and reported observations are applicable to other scanner types, since the proposed method operates directly in the image space. Next, we recover the images using the regularized least squares algorithm (RLS), which is commonly adopted in CT reconstruction. We emulate the limited-angle scenario by providing only a partial sinogram to RLS. We provide the ing reconstruction as the input to the proposed algorithm. Experimental Settings: On both datasets, we train a standard DCGAN to generate images using the 60K training 28 × 28 images. We run all our reconstruction experiments on a subset of the 10K validation set. Corruption-mimicking requires choosing 4 main hyperparameters: T 1 = 15, T 2 = 15, γ s = 1e − 2, γ g = 8e − 2 that control the number of iterations in the alternating optimization and learning rates (see section 2 for details), these are kept fixed on both datasets, across all viewing angle settings. We observed the performance to be robust across a wide range of settings for these hyper-parameters. Finally, we compare the performance of the robust GAN prior against the standard GAN prior, without corruption-mimicking. In both cases, we run the latent space optimization for a total of ∼ 2500 iterations, which typically only takes about 10 seconds on a P100 NVIDIA GPU. In figures 1, 2, we show qualitative and quantitative obtained for both the MNIST and Fashion-MNIST datasets respectively. In both cases, we demonstrate significant improvements in recovering the true reconstruction compared to the vanilla GAN prior. It should be noted that a performance boost of nearly 4-5 dB on MNIST and 0.5-1dB on Fashion-MNIST are achieved with no additional information or data, but due to the inclusion of the robust GAN prior. Additionally, PSNR and SSIM tend to be uncorrelated with perceptual metrics in many cases, as perceptually poor reconstructions can be deceptively close in PSNR or SSIM. A potential fix in GAN-based reconstruction approaches is to compute error in the discriminator feature space as a proxy for perceptual quality.: Given the RLS reconstruction, we improve them by projecting onto the image manifold using corruption mimicking. In all cases, we show the improvement obtained by using the robust GAN prior over a standard GAN projection.

We show that robust GAN priors work better than GAN priors for limited angle CT reconstruction which is a highly under-determined inverse problem.

Represent the TLDR for finding the corresponding scientific article


Question: We propose an effective multitask learning setup for reducing distant supervision noise by leveraging sentence-level supervision. We show how sentence-level supervision can be used to improve the encoding of individual sentences, and to learn which input sentences are more likely to express the relationship between a pair of entities. We also introduce a novel neural architecture for collecting signals from multiple input sentences, which combines the benefits of attention and maxpooling. The proposed method increases AUC by 10% (from 0.261 to 0.284), and outperforms recently published on the FB-NYT dataset. Early work in relation extraction from text used fully supervised methods, e.g., BID2, which motivated the development of relatively small datasets with sentence-level annotations such as ACE 2004 BID2, BioInfer and SemEval 2010. Recognizing the difficulty of annotating text with relations, especially when the number of relation types of interest is large, BID16 pioneered the distant supervision approach to relation extraction, where a knowledge base (KB) and a text corpus are used to automatically generate a large dataset of labeled sentences which is then used to train a relation classifier. Distant supervision provides a practical alternative to manual annotations, but introduces many noisy examples. Although many methods have been proposed to reduce the noise in distantly supervised models for relation extraction (e.g., BID8 BID23 BID22 BID5 BID27 BID11, a rather obvious approach has been understudied: using sentence-level supervision to augment distant supervision. Intuitively, supervision at the sentence-level can help reduce the noise in distantly supervised models by identifying which of the input sentences for a given pair of entities are likely to express a relation. We experiment with a variety of model architectures to combine sentence-and bag-level supervision and find it most effective to use the sentence-level annotations to directly supervise the sentence encoder component of the model in a multi-task learning framework. We also introduce a novel maxpooling attention architecture for combining the evidence provided by different sentences where the entity pair is mentioned, and use the sentence-level annotations to supervise attention weights. The contributions of this paper are as follows:• We propose an effective multitask learning setup for reducing distant supervision noise by leveraging existing datasets of relations annotated at the sentence level.• We propose maxpooled attention, a neural architecture which combines the benefits of maxpooling and soft attention, and show that it helps the model combine information about a pair of entities from multiple sentences.• We release our library for relation extraction as open source. 1 The following section defines the notation we use, describes the problem and provides an overview of our approach. Our goal is to predict which relation types are expressed between a pair of entities (e 1, e 2), given 1 We attach the anonymized code as supplemental material in this submission instead of providing a github link in order to maintain author anonymity. Figure 1: An overview of our approach for augmenting distant supervision with sentence-level annotations. The left side shows one sentence in the labeled data and how it is used to provide direct supervision for the sentence encoder. The right side shows snippets of the text corpus and the knowledge base, which are then combined to construct one training instance for the model, with a bag of three input sentences and two active relations:'founder of' and'ceo of'.all sentences in which both entities are mentioned in a large collection of unlabeled documents. Following previous work on distant supervision, we use known tuples (e 1, r, e 2) in a knowledge base K to automatically annotate sentences where both entities are mentioned. In particular, we group all sentences s with one or more mentions of an entity pair e 1 and e 2 into a bag of sentences B e 1,e 2, then automatically annotate this bag with the set of relation types L distant = {r ∈ R : (e 1, r, e 2) ∈ K}, where R is the set of relations we are interested in. We use'positive instances' to refer to cases where |L| > 0, and'negative instances' when |L| = 0.In this paper, we leverage existing datasets with sentence-level relation annotations in a similar domain, where each example consists of a token sequence s, token indexes for e 1 and e 2 in the sequence, and one relation type (or 'no relation'). Since the relation types annotated at the sentence level may not correspond one-to-one to those in the KB, we replace the relation label associated with each sentence with a binary indicator. (1 indicates that the sentence s expresses one of the relationships of interest.) We do not require the entities to match those in the KB either. Fig. 1 illustrates how we modify neural architectures commonly used in distant supervision, e.g., BID13 to effectively incorporate sentence-level supervision. The model consists of two components: 1) A sentence encoder (displayed in blue) reads a sequence of tokens and their relative distances from e 1 and e 2, and outputs a vector s representing the sentence encoding, as well as P (e 1 ∼ e 2 | s) representing the probability that the two entities are related given this sentence. 2) The bag encoder (displayed in green) reads the encoding of each sentence in the bag for the pair (e 1, e 2) and predicts P (r = 1 | e 1, e 2), ∀r ∈ R. We combine both bag-level (i.e., distant) and sentence-level (i.e., direct) supervision in a multi-task learning framework by minimizing the weighted sum of the cross entropy losses for P (e 1 ∼ e 2 | s) and P (r = 1 | e 1, e 2). By sharing the parameters of sentence encoders used to compute either loss, the sentence encoders become less susceptible to the noisy bag labels. The bag encoder also benefits from sentence-level supervision by using the supervised distribution P (e 1 ∼ e 2 | s) to decide the weight of each sentence in the bag, using a novel architecture which we call maxpooled attention. The model predicts a set of relation types L pred ⊂ R given a pair of entities e 1, e 2 and a bag of sentences B e 1,e 2. In this section, we first describe the sentence encoder part of the model FIG2, then describe the bag encoder FIG2, then we explain how the two types of supervision are jointly used for training the model end-to-end. Given a sequence of words w 1,..., w |s| in a sentence s, a sentence encoder translates this sequence into a fixed length vector s. Input Representation. The input representation is illustrated graphically with a table at the bottom of FIG2. We map word token i in the sentence w i to a pretrained word embedding vector w i. 2 Another crucial input signal is the position of entity mentions in each sentence s ∈ B e 1,e 2. Following BID28, we map the distance between each word in the sentence and the entity mentions 3 to a small vector of learned parameters, namely d i. Instead of randomly initializing position embeddings with mean = 0, we obtain notable performance improvements by randomly initializing all dimensions of the position embedding for distance d around the mean value d. Intuitively, this makes it easier to learn useful parameters since the embedding of similar distances (e.g., d = 10 and d = 11) should be similar, without adding hard constraints on how they should be related. We find that adding a dropout layer with a small probability (p = 0.1) before the sentence encoder reduces overfitting and improves the . To summarize, the input layer for a sentence s is a sequence of vectors: DISPLAYFORM0 Word Composition. Word composition is illustrated with the block CNN in the bottom part of FIG2, which represents a convolutional neural network (CNN) with multiple filter sizes. The outputs of the maxpool operations for different filter sizes are concatenated then projected into a smaller vector using one feed forward linear layer. This is in contrast to previous work BID19 which used Piecewise CNN (PCNN). In PCNN, we convolve three segments of the sentence separately: windows before the left entity, windows inbetween the two entities and windows after the right entity. Every split is maxpooled independently, then the three vectors are concatenated. The intuition is that this helps the model put more emphasis on the middle segment which: Blue box is the sentence encoder, it maps a sentence to a fixed length vector (CNN output) and the probability it expresses a relation between e 1 and e 2 (sigmoid output). Green box is the bag encoder, it takes encoded sentences and their weights and produces a fixed length vector (maxpool output), concatenates it with entity embeddings (pointwise mult. output) then outputs a probability for each relation type r. White boxes contain parameters that the model learns while gray boxes do not have learnable parameters. Sentence-level annotations supervise P (e 1 ∼ e 2 | s). Bag-level annotations supervise P (r = 1 | e 1, e 2).connects the two entities. As discussed later in Section 4.2, we compare CNN and PCNN and find the simpler CNN architecture works better. Sentence encoding s is computed as follows: DISPLAYFORM1 where CNN x is a standard convolutional neural network with filter size x, W 1 and b 1 are model parameters and s is the sentence encoding. We feed the sentence encoding s into a ReLU layer followed by a sigmoid layer with output size 1, representing P (e 1 ∼ e 2 | s), as illustrated in DISPLAYFORM2 where σ is the sigmoid function and W 2, b 2, W 3, b 3 are model parameters. Given a bag B e 1,e 2 of n ≥ 1 sentences, we compute their encodings s 1,..., s n as described earlier and feed them into the bag encoder, which is responsible for combining the information in all sentence encodings and predict the probability P (r = 1 | e 1, e 2), ∀r ∈ R. The bag encoder also makes use of p = P (e 1 ∼ e 2 | s) from Eq. 1 as an estimate of the degree to which sentence s expresses the relation between e 1 and e 2.Maxpooled Attention. To aggregate the sentence encodings s 1,..., s n into a fixed length vector that captures the important features in the bag, BID11 used maxpooling, while BID13 used soft attention. In this work, we propose maxpooled attention, a new form of attention which combines some of the characteristics of maxpooling and soft attention. Given the encoding s j and an unnormalized weight u j for each sentence s j ∈ B e 1,e 2, the bag encoding g is a vector with the same dimensionality as s j with the k-th element computed as: DISPLAYFORM0 Maxpooled attention has the same intuition of soft attention; learning weights for sentences that enable the model to focus on the important sentences. However, maxpooled attention differs from soft attention in two aspects. The first is that every sentence s j is given a probability that indicates how useful the sentence is, independently of the other sentences. Notice how this is different from soft attention where sentences compete for probability mass, i.e., probabilities must sum to 1. This is implemented in maxpooled attention by normalizing the weight of each sentence with a sigmoid function rather than a softmax. This is a better fit for the task at hand because the sentences are not competing. It also makes the weights useful even when |B e 1,e 2 | = 1, while soft attention will always normalize such weights to 1.The second difference between maxpooled attention and soft attention is the use of weighted maxpooling instead of weighted average. Maxpooling is more effective for this task because it can pick the useful features from different sentences. As shown in FIG2, we do not directly use the p from Eq. 1 as weight in maxpooled attention. Instead, we found it useful to feed it into more non-linearities. The unnormalized maxpooled attention weight for s j is computed as: DISPLAYFORM1 Entity Embeddings. , we use entity embeddings to improve our model of relations in the distant supervision setting, although our formulation is closer to that of BID25 who used point-wise multiplication of entity embeddings: m = e 1 e 2, where is point-wise multiplication, and e 1 and e 2 are the embeddings of e 1 and e 2, respectively. In order to improve the coverage of entity embeddings, we use pretrained GloVe vectors BID19 (same embeddings used in the input layer).For entities with multiple words, like "Steve Jobs", the vector for the entity is the average of the GloVe vectors of its individual words. If the entity is expressed differently across sentences, we average the vectors of the different mentions. As discussed in Section 4.2, this leads to big improvement in the , and we believe there's still big room for improvement from having better representation for entities. We feed the output m as additional input to the last block of our model. Output Layer. The final step is to use the bag encoding g and the entity pair encoding m to predict a set of relations L pred which is a standard multilabel classification problem. We concatenate g and m and feed them into a feedforward layer with ReLU non-linearity, followed by a sigmoid layer with an output size of |R|: DISPLAYFORM2 where r is a vector of Bernoulli variables each of which corresponds to one of the relations in R. This is the final output of the model. To train the model on the bag-level labels obtained with distant supervision, we use binary crossentropy loss between the model predictions and the labels obtained with distant supervision, i.e., bag loss = DISPLAYFORM0 where r distant [k] = 1 indicates that the tuple (e 1, r k, e 2) is in the knowledge base. In addition to the bag-level supervision commonly used in distant supervision, we also use sentence-level annotations. One approach is to create a bag of size 1 for each sentence-level annotation, and add the bags to those obtained with distant supervision. However, this approach requires mapping relations in the sentence-level annotations map to those in the KB.Instead, we found that the best use of the supervised data is to improve the model's ability to predict the the potential usefulness of a sentence by using sentence-level annotations to help supervise the sentence encoder module. According to our analysis of baseline models, distinguishing between positive and negative examples is the real bottleneck. This supervision serves two purposes: it improves our encoding of each sentence, and improves the weights used by the maxpooled attention to decide which sentences should contribute more to the bag encoding. We maximize log loss of gold labels in the sentence-level data D according to the model described in Eq. 1: DISPLAYFORM1 where D consists of all the sentence-level annotations in addition to all distantly-supervised negative examples. 4 We jointly train the model on both types of supervision. The model loss is a weighted sum of the sentence-level and the bag-level losses: DISPLAYFORM2 where λ is a parameter that controls the contribution of each loss, tuned on a validation set. This section discusses datasets, metrics, experiment configurations and the models we are comparing with. Metrics. Prior works used precision-recall (PR) curves to show their on the FB-NYT dataset. In this multilabel classification setting, all model predictions for all relation types are sorted by confidence from highest to lowest. Then applying different thresholds gives the points on the PR curve. We use the area under the PR curve (AUC) for early stopping and hyperparameter tuning. Because we are interested in the high-precision extractions, we focus on the high-precision lowrecall part of the PR curve. That is, in our experiments, we only keep points on the PR curve with recall below 0.4 which means that the largest possible value for AUC is 0.4.Configurations. The FB-NYT dataset does not have a validation set for hyper-parameter tuning and early stopping. For these, use the test set, and BID13 use 3-fold cross validation. We use 90% of the training set for training and keep the other 10% for validation. The pretrained word embeddings we use are 300-dimensional GloVe vectors, trained on 42B tokens. Since we do not update word embeddings 6 while training the model, our vocabulary may include any word which appears in the training, validation or test sets with frequency greater than two. When a word with a hyphen (e.g., 'five-star') is not in the GloVe vocabulary, we average the embeddings of its subcomponents. Otherwise, all OOV words are assigned the same random vector (normal with mean 0 and standard deviation 0.05).Our model is implemented using PyTorch and AllenNLP BID6 and trained on machines with P100 GPUs. Each run takes five hours on average. We train for a large number of epochs and use early stopping with patience = 3. The batches of the two datasets are randomly shuffled before every epoch. The optimizer we use is Adam with its default PyTorch parameters. We run every configuration three times with three different random seeds then report the PR curve for the run with the best validation (thresholded) AUC. In the controlled experiments, we report the mean and standard deviation of the AUC metric. Compared Models. Our baseline for comparison is a model that is similar to what is described in Section 3 with the following configurations. It uses our approach for position embedding initialization, encodes sentences using CNN, uses entity embeddings, aggregate sentences using maxpooling and does not use the sentence-level annotation. Our best configuration adds the maxpooled attention and the sentence-level annotations. We also compare with existing models in the literature. The model by BID13 uses an attention mechanism that assigns weights to each sentence followed by a weighted average of sentence encodings. The model by extends the model by BID13 by using soft labels during training. Main Result. FIG3 summarizes the main of our experiments. 8 The AUC of our baseline (green) is comparable to that of (blue), which verifies that we are building on a strong baseline. Adding maxpooled attention and sentence-level supervision (i.e., the full model, in red) substantially improves over the baseline (green). The figure also illustrates that our full model outperforms the strong baseline 8 Results of BID13 and are copied from their papers. 0.271 ± 0.007 + additional bags 0.269 ± 0.001 + sentence loss 0.284 ± 0.007 Table 1: The + and − signs indicate independent changes to the baseline configuration.of in orange. 9 We emphasize that the improved reported here conflate both additional supervision and model improvements. Next, we report the of controlled experiments to quantify the contribution of each modification in Table 1. The first line in the table is the baseline model and configuration described in the previous section and in FIG3, and the + and − signs indicate (independent) additions to and removals from that configuration, respectively. Position Embedding Initialization. The second line in Table 1 shows that removing the distance-based initialization of position embeddings in a large drop in AUC. We hypothesize that the position-based initialization reduces the burden of finding optimal values for position embeddings, without explicit constraints that guarantee similar distances to have similar embeddings. Sentence Encoder. In the next line of Table 1, we replace the simpler CNN in our baseline with the more complex PCNN BID27. Both encoders use filters of sizes 2, 3, 4 and 5. Table 1 shows that using CNN works markedly better than PCNN which is in contrast to the findings of BID27. This could be due to the use of multiple filter sizes and to the improved representation of entity positions in our model, which may obviate the need to have a separate encoding of each segment in the sentence. Entity Embeddings. The next line in Table 1 shows that entity embeddings (which are included in the baseline model) provide valuable information and help predict relations. This information may encode entity type, entity compatibility with each others and entity bias to participate in a relation. Given that our entity embeddings are simple GloVe vectors, we believe there is still a large room for improvement. We compare different ways of aggregating sentences into a single vector including maxpooling (baseline, originally proposed in BID11), attention BID13 and our proposed maxpooled attention. 10 Maxpooling works better than soft attention because it is better at picking out useful features from multiple sentences, while attention can only weight the whole representation of the sentence. We hypothesize that our proposed maxpooled attention works better than both because it combines the soft attention's ability to learn and use different weights for different sentences, and the maxpool's ability to pick out useful features from multiple sentence. Another advantage of maxpooled attention over attention is that it helps in cases where bag size equals 1 because the softmax typically used in attention in a weight of 1 for the sentence rendering that weight useless. The last three lines in Table 1 compare different ways for using sentence-level annotations. The line "baseline + maxpooled att." is copied from the pre-10 Our reimplementation of BID13 attention differs from what was described in the paper. The unnormalized attention weights of BID13 are oj = sj × A × q where sj is the sentence encoding, A is a diagonal matrix and q is the query vector. We tried this but found that implementing it as a feedforward layer with output size = 1 works better. The in Table 1 are for the feedforward implementation. vious line and is the basis for the following two lines. In "additional bags," we add the sentencelevel annotations as additional bags along with the distantly supervised data. In "sentence loss," we use the method described in Section 3 for integrating sentence-level supervision. The show that simply adding the sentence-level supervised data to the distantly supervised data as additional bags has little effect on the performance. This is probably because they change the distribution of the training to differ from the test set. However, adding the sentence-level supervision following our proposed multitask learning improves the considerably because it allows the model to better filter noisy sentences. Selecting Lambda. Although we did not spend much time tuning hyperparameters, we made sure to carefully tune λ (Equation 3) which balances the contribution of the two losses. Early experiments showed that sentence-level loss is typically smaller than bag-level loss, so we experimented with λ ∈ {0, 0.5, 1, 2, 4, 8, 16, 32, 64}. FIG4 shows thresholded AUC for different values of λ, where each point is the average of three runs. It is clear that picking the right value for λ has a big impact on the final . Qualitative Analysis. An example of a positive bag is shown in TAB3. Our model, which incorporates sentence-level supervision, assigns the most weight to the first sentence while the attention model assigns the most weight to the last sentence (which is less informative for the relation between the two entities). Furthermore, the attention model does not use the other two sentences because their weights are dominated by the weight of the last sentence. We also found that the weights from our model usually range between 0 and 0.08, suggesting the relative values of the weights are informative to the model, even when the absolute values are small. Distant Supervision. The term'distant supervision' was coined by BID16 who used relation instances in a KB to identify any sentence in a text corpus where two related entities are mentioned, then developed a classifier to predict the relation. Researchers have since extended this approach for relation extraction (e.g., BID24 BID15 BID21 BID12 . A key source of noise in distant supervision is that sentences may mention two related entities without expressing the relation between them. BID8 used multi-instance learning to address this problem by developing a graphical model for each entity pair which includes a latent variable for each sentence to explicitly indicate the relation expressed by that sentence, if any. Our model can be viewed as an extension of BID8 where the sentence-bound latent variables can also be directly supervised in some of the training examples. Neural Models for Distant Supervision. More recently, neural models have been effectively used to model textual relations (e.g., BID7 BID28 BID17 . Focusing on distantly supervised models, BID27 proposed a neural implementation of multi-instance learning to leverage multiple sentences which mention an entity pair in distantly supervised relation extraction. However, their model picks only one sentence to represent an entity pair, which wastes the information in the neglected sentences. BID11 addresses this limitation by maxpooling the vector encodings of all input sentences for a given entity pair. BID13 independently proposed to use attention to address the same limitation. Results in Section 4.2 suggest that maxpooling is more effective than attention for multi-instance learning. BID26 proposed a method for leveraging dependencies between different relations in a pairwise ranking framework. Sentence-Level Supervision. Despite the substantial amount of work on both fully supervised and distantly supervised relation extraction, the question of how to combine both signals has been mostly ignored in the literature, with a few exceptions. first manually defined a mapping between relation types in YAGO to compatible relation types in ACE 2004 BID4, then trained two separate SVM models using the training portion of ACE 2004 and the distantly supervised sentences. Model predictions are then linearly combined to make the final prediction. In contrast, we use a neural model which combines both sources of supervision in a multi-task learning framework BID3 . We also do not require a strict mapping between the relation types of the KB and those annotated at the sentence level. Another important distinction is the unit of prediction (at the sentence level vs. at the entity pair level), each of which has important practical applications. Also related is BID0 who used active learning to improve the multi-instance multi-label model of BID23. We propose two complementary methods to improve performance and reduce noise in distantly supervised relation extraction. The first is incorporating sentence-level supervision and the second is maxpooled attention, a novel form of attention. The sentence-level supervision improves sentence encoding and provides supervision for attention weights, while maxpooled attention effectively combines sentence encodings and their weights into a bag encoding. Our experiments show a 10% improvement in AUC (from 0.261 to 0.284) outperforming recently published on the FB-NYT dataset.

Represent the TLDR for finding the corresponding scientific article

A new form of attention that works well for the distant supervision setting, and a multitask learning approach to add sentence-level annotations.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.