Question: represent the text to find a very short summary (1-2 sentences max)

Question: Despite a growing literature on explaining neural networks, no consensus has been reached on how to explain a neural network decision or how to evaluate an explanation. 	 Our contributions in this paper are twofold. First, we investigate schemes to combine explanation methods and reduce model uncertainty to obtain a single aggregated explanation. The aggregation is more robust and aligns better with the neural network than any single explanation method.. 	 Second, we propose a new approach to evaluating explanation methods that circumvents the need for manual evaluation and is not reliant on the alignment of neural networks and humans decision processes. Despite the great success of neural networks especially in classic visual recognition problems, explaining the networks' decisions remains an open research problem. This is due in part to the complexity of the visual recognition problem and in part to the basic'ill-posedness' of the explanation task. This challenge is amplified by the fact that there is no agreement on what a sufficient explanation is and how to evaluate an explanation method. Many different explanation strategies and methods have been proposed (; ; ; ; ;). Focusing on visual explanations for individual decisions, most methods either use a backpropagation approach or aim to construct a simpler linear model with an intuitive explanation. The plethora of explanation approaches is a signature of the high-level epistemic uncertainty of the explanation task. This paper is motivated by a key insight in machine learning: Ensemble models can reduce both bias and variance compared to applying a single model. A related approach was pursued for functional visualization in neuroimaging . Here we for the first time explore the potential of aggregating explanations of individual visual decisions in reducing epistemic uncertainty for neural networks. We test the hypothesis that ensembles of multiple explanation methods are more robust than any single method. This idea is analyzed theoretically and evaluated empirically. We discuss the properties of the aggregate explanations and provide visual evidence that they combine features, hence are more complete and less biased than individual schemes. Based on this insight, we propose two ways to aggregate explanation methods, AGG-Mean and AGG-Var. In experiments on Imagenet, MNIST, and FashionMNIST, the aggregates identify relevant parts of the image more accurately than any single method. Second, we introduce IROF (Iterative Removal Of Features) as a new approach to quantitatively evaluate explanation methods without relying on human evaluation. We circumvent the problems of high correlation between neighbor pixels as well as the human bias that are present in current evaluation methods. The open problem of explainability is reflected in a lot of recent work (; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;). We focus on generating visual explanations for single samples. The first work in this direction was with Saliency Maps (SM) that proposed backpropagating the output onto the input to gain an understanding of a neural network decision. The relevance for each input dimension is extracted by taking the gradient of the output w. r. t. to the input. This idea was extended by into Guided Backpropagation (GM) by applying ReLU non-linearities after each layer during the backpropagation. Compared to Saliency, this removes visual noise in the explanation. Grad-CAM (GC) from is an explanation method, developed for use with convolutional neural networks. By backpropagating relevance through the dense layers and up-sampling the evidence for the convolutional part of the network, the method obtains coarse heatmaps that highlight relevant parts of the input image. Integrated Gradients (IG) sums up the gradients from linearly interpolated pictures between a baseline, e.g. a black image, and the actual image. SmoothGrad (SG) filters out noise from a basic saliency map by creating many samples of the original input with Gaussian noise. The final saliency map is the average over all samples. Finally, we also consider. In contrast to the other methods, LIME is not based on backpropagation. Instead, it approximates the neural network with a linear model locally around the input to be explained. The coefficients of the linear model for the respective input dimensions give the importance of each dimension. Compared to the other methods this is much more computationally expensive as it requires many passes through the neural network. The evaluation of explanation methods is a relatively recent topic with few systematic approaches (; ; ; ;). To our knowledge, proposed the first quantitative approach to evaluate an explanation method by flipping pixels to their opposite and comparing the decrease in output with the relevance attributed to the pixel by the explanation method. As the authors note, this only works for low-dimensional input. This work was followed up upon in. By dividing high-dimensional images into squares, they make the method feasible for high-dimensional inputs. Squares with high relevance (as measured by the explanation method) consecutively get replaced with noise sampled from the uniform distribution. The difference between the original output and the output for the degraded images indicates the quality of the explanation method. proposes another quantitative approach to evaluate explanation methods called ROAR. For each explanation method they extract the relevance maps over the entire training set. They degrade the training set by setting different percentages of the pixels with the highest relevance to the mean and retrain the network. Each retrained network is evaluated on the test set. The accuracy on the test set decreases dependent on the percentage of pixels set to the mean. This requires retraining the same architecture multiple times for each explanation method at a high computational cost. proposed a different approach to evaluate explanation methods, called Sensitivityn, based on the notion that the decrease in output when a number of inputs are canceled out should be equal to the sum of their relevances. For a range of n (between 1 and the total number of inputs) they sample a hundred subsets of the input. For each n, the Pearson Correlation Coefficient (PCC) between the decrease in output, when the subset of features is removed, and the sum of their relevances is reported. The is a curve of the PCC dependent on the percentage of the input being removed. For a good explanation method, the PCC will decrease slowly. All currently available explanation methods have weaknesses that are inherent to the approach and include significant uncertainty in the ing heatmap (; ;). A natural way to mitigate this issue and reduce noise is to combine multiple explanation methods. Ensemble methods have been used for a long time to reduce the variance and bias of machine learning models. We apply the same idea to explanation methods and build an ensemble of explanation methods. We assume a neural network F: X → y with X ∈ R m×m and a set of explanation methods {e j} J j=1 with e j: X, y, F → E with E ∈ R m×m. We write E j,n for the explanation obtained for X n with method e j and denote the mean aggregate explanation asē withĒ n = 1 J J j=1 E j,n. While we assume the input to be an image ∈ R m×m, this method is generalizable to inputs of other dimensionalities as well. To get a theoretical understanding of the benefit of aggregation, we hypothesize the existence of a'true' explanationÊ n. This allows us to quantify the error of an explanation method as the mean squared difference between the'true' explanation and an explanation procured by an explanation method, i.e. the MSE. For clarity we subsequently omit the notation for the neural network. We write the error of explanation method j on image X n as err j,n = ||E j,n −Ê n || 2 with is the MSE of the aggregate. The typical error of an explanation method is the mean error over all explanation methods With these definitions we can do a standard bias-variance decomposition . Accordingly we can show the error of the aggregate will be less that the typical error of explanation methods, A detailed calculation is given in appendix A.1. The error of the aggregate MSE(Ē) is less than the typical error of the participating methods. The difference -a'variance' term -represents the epistemic uncertainty and only vanishes if all methods produce identical maps. By taking the average over all available explanation methods, we reduce the variance of the explanation compared to using a single method. To obtain this average, we normalize all input heatmaps such that the relevance over all pixels sum up to one. This reflects our initial assumption that all individual explanation methods are equally good estimators. We refer to this approach as AGG-Mean. size' map by dividing the mean aggregate locally with its standard deviation . Intuitively, this will assign less relevance to segments with high disagreement between methods. For stability, we divide not directly by the local variance but add a constant to the estimate of the local variance. This can be interpreted as a smoothing regularizer or a priori information regarding epistemic and aleatoric uncertainties. We refer to this approach as AGG-Var. where σ(E j∈J,n) is the point-wise standard deviation over all explanations j ∈ J for X n In section 4 we will evaluate and compare AGG-Mean and AGG-Var against basic explanation methods. Figure 1: Quantitative evaluation with IROF: Relevant segments as identified by an explanation method get consecutively replaced by the mean colour over the entire dataset. The IROF score of an explanation method is the integrated decrease in the class score over the number of removed segments. Quantitative evaluation is a recurring problem with explainability methods. This is especially true for high-dimensional input, such as images, where important features are made up of locally highly correlated pixels. If the information in one pixel is lost, this will not change the overall feature and should therefore not in a changed output score. The relevance values of single pixels are not indicative of the feature's importance as a whole. We circumvent this problem by utilizing conventional image segmentation, a well-explored area. By first dividing the image into coherent segments, we avoid the interdependency between the inputs. Methodology We assume a neural network F: X → y with X ∈ R m×m and a set of explanation methods {e j} J j=1 with e j: X, y, F → E with E ∈ R m×m. Furthermore we partition each image X n into a set of segments {S l n} L l=1 using a given segmentation method with s l n,i,j = 1 indicating that pixel x n,i,j belongs to segment l. Computing the mean importance of each segment according to a given explanation method j, two segments can be directly compared against each other. By sorting the segments in decreasing order of importance according to the explanation method, we get a ranking of how relevant each segment of the image is. We use X l n to indicate X n with the l segments with highest mean relevance replaced with the mean value. Taking F (X l n) y repeatedly with increasing l ∈ 0,..., L in a curve of the class score dependent on how many segments of the image are removed. Dividing this curve by F (X 0 n) y normalizes the scores to be within and makes curves comparable between input samples and networks. If an explanation method works well, it will attribute high relevance to segments important for classification. As segments with high relevance are removed first, the score for the target class will go down faster. By computing the area over the curve (AOC) for the class score curve and averaging over a number of input samples, we can identify the method that identifies relevant areas of the image more reliably. For a good explanation method, the AOC will be higher. We refer to this evaluation method as the iterative removal of features (IROF). The IROF for a given explanation method e j is expressed as: This approach is a quantitative comparison of two or more explainability methods that does not rely on human evaluation or alignment between human and neural network reasoning. For each explanation method the workflow produces a single value, enabling convenient comparison between two or more explanation methods. If the AOC is higher, the explanation method captures more information about the neural network classification. IROF is dependent on having meaningful segments in the input, as natural images do. Dividing up text or non-natural images such as EEG into meaningful and independent segments does not have a natural solution and is left for future research. We first present empirical validation of our proposed evaluation technique IROF in section 4.2. Subsequently we evaluate the aggregation of explanation techniques against the vanilla techniques with IROF, Sensitivity-n and qualitative evaluation. In appendix A.6.1 we compare aggregated methods on a dataset of human-annotated heatmaps. We tested our method on five neural network architectures that were pre-trained on ImageNet: VGG19, Xception, Inception, ResNet50 and ResNet101;;;. 1 Additionally, we ran experiments on CNN trained on the MNIST and FashionMNIST dataset;. We compared the aggregation methods against Saliency (SM), Guided Backpropagation (GB), SmoothGrad (SG), Grad-CAM (GC) and Integrated Gradients (IG) to have a selection of attributionbased methods. Additionally we compared against LIME as a method that is not based on attribution but rather on local approximation. The aggregations are based on all attributionbased methods. Some of the methods in positive and negative evidence. We only considered positive evidence for the ImageNet tasks to compare methods against each other. To check that this does not corrupt the methods, we compared the methods that do contain negative against their filtered version and found negligible difference between the two versions of a method in the used metrics. For Agg-Mean we introduced an additional parameter, to the divisor. In our experiments we set to be ten times the mean σ over the entire dataset. A good evaluation method should be able to reject the null hypothesis (a given explanation method is no better than random choice) with high confidence. We use this as motivation to evaluate and compare IROF by calculating the paired t-test of an explanation method versus random guessing. This is done with multiple explanation methods and networks, to reduce the impact of the explanation method. We compare IROF and pixel removal with mean value and black as a replacement value respectively. Additionally we compare against as explained in section 2. we set the 10% most relevant segments to the mean value over the dataset. For pixel removal, we set the equivalent number of pixels to the mean value. The percentage of segments or pixels being removed was chosen ad hoc. If the difference in degradation between random choice and the explanation method is high, the explanation method reports meaningful information. Since we compare the same explanation method on the same neural network with different evaluation methods, the p-values only contain information about how meaningful the evaluation method is. We computed IROF and pixel removal with black or mean replacement values and compared the p-value changes dependent on the number of samples. Results are shown in fig. 2 (extended in appendix A.6). In table 1 we provide for forty images in tabular form for an easier overview (other methods in appendix A.6). On forty images, all evaluation methods produce p-values below 0.05. Thus, all evaluation methods can distinguish between random guessing and an explanation method. However, IROF can reject the null hypothesis (the explanation method does not contain any information), with much higher confidence with the same number of samples for any configuration. We can conclude that IROF is more sensitive to the explanation method than pixel removal or , making it the better choice to quantitatively evaluate an explanation method. the IROF as described in section 3.2. We include two non-informative baselines. Random randomly chooses segments to remove. Sobel is a sobel edge detector. Neither of them contain information about the neural network. All explanation methods have a lower IROF than the random baseline on all architectures tested, indicating that all methods contain information about the image classification. Except for LIME, all methods also surpass the stronger baseline, SOBEL. The ranking of unaggregated methods varies considerably between architectures. This variance indicates that the accuracy of an explanation method depends on the complexity and structure of the neural network architecture. For all architectures AGG-Mean and AGG-Var have a lower IROF score than any non-aggregated method. For ResNet101 the difference between the best unaggregated method and the aggregated methods is especially large. We hypothesize, that the benefit of aggregating explanation methods increases for more complex neural network with large epistemic uncertainty on the explanation. We can empirically confirm that aggregating methods improves over unaggregated methods and more reliably identifies parts of the images that are relevant for classification. We show heatmaps for individual examination for each of the methods in fig. 3 and compare qualitatively (large version of fig. 3 in appendix A.6.2). While visual evaluation of explanations for neural networks can be misleading, there is no better way available of checking whether any given explanation method agrees with intuitive human understanding . Additionally, we compute alignment between human-annotated images and the explanation methods in??, using the human benchmark for evaluation introduced in. AGG-Var combines features of the aggregrated methods by attributing relevance on the classified object as a whole, but considering smaller details such as the face of an animal as more relevant. It is a combination of the detail-oriented and shape-oriented methods. Compared to SmoothGrad, which concentrates on one isolated feature, the relevance is more evenly distributed and aligned with our human intuition that classification takes context into account and does not rely on e.g. only the beak of a bird. We can conclude that combining explainability methods provides a meaningful visual improvement over single methods. To quantitatively compare explanation methods on a low-dimensional input we use Sensitivity-n . The exact procedure is described in section 2. We compare on MNIST and FashionMNIST , two low-dimensional dataset with a basic CNN 2 (architecture in appendix). We follow the procedure suggested in and test on a hundred randomly sampled subsets for 1000 randomly sampled test images. The number of pixels in the set n is chosen at fifteen points logarithmically spaced between 10 and 780 pixels. As described in section 2, for a range of n (between 1 and the total number of inputs) a hundred subsets of the input features are removed. For each n, the average Pearson Correlation Coefficient (PCC) between the decrease in output and the relevance of the removed output features is reported. The is a curve of the PCC dependent on the removed percentage. We show in fig. 4. AGG-Mean and AGG-Var perform in range of the best methods. For the CNN trained on FashionMNIST, AGG-Mean and AGG-Var perform better than unaggregated methods. For the CNN trained on MNIST, Guided Backpropagation and AGG-Mean perform best. For both networks (trained on FashionMNIST and MNIST respectively), SmoothGrad and GradCAM perform considerably worse than the other methods. In summary, aggregation seems to not be as beneficial when applied to a low-dimensional, "easier" tasks such as MNIST as it is for ImageNet, performing in range of the best unaggregated method. We hypothesize that this is because there is less epistemic uncertainty in explanations for less complex tasks and network architectures. In this work we gave a simple proof that aggregating explanation methods will perform at least as good as the typical individual method. In practice, we found evidence that aggregating methods outperforms any single method. We found this evidence substantiated across quantitative metrics. While our show that different vanilla explanation methods perform best on different network architectures, an aggregation supersedes all of them on any given architecture. Additionally we proposed a novel way of evaluation for explanation methods that circumvents the problem of high correlation between pixels and does not rely on visual inspection by humans, an inherently misleading metric. All currently available explanation methods have weaknesses that are inherent to the approach and include significant noise in the heatmap (; ;). A natural way to mitigate this issue and reduce noise is to combine multiple explanation methods. Ensemble methods have been used for a long time to reduce the variance and bias of machine learning models. We apply the same idea to explanation methods and build an ensemble of explanation methods. We assume a neural network F: X → y with X ∈ R mxm and a set of explanation methods {e j} J j=1 with e j: X, y, F → E with E ∈ R mxm. We write E j,n for the explanation obtained for X n with method e j and denote the mean aggregate explanation asē withĒ n = 1 J J j=1 E j,n. While we assume the input to be an image ∈ R mxm, this method is generalizable to inputs of other dimensions as well. We define the error of an explanation method as the mean squared difference between a hypothetical'true' explanation and an explanation procured by the explanation method, i.e. the MSE. For this definition we assume the existence of the hypothetical'true' explanationÊ n for image X n. For clarity we subsequently omit the notation for the neural network. We write the error of explanation method j on image X n as err j,n = ||E j,n −Ê n || 2 with and MSE(Ē) = 1 N n ||Ē n −Ê n || 2 is the MSE of the aggregate. The typical error of an explanation method is represented by the mean The error of the aggregate MSE(Ē) is less than the typical error of the participating methods. The difference -a'variance' term -represents the epistemic uncertainty and only vanishes if all methods produce identical maps. In section 3.1 we showed theoretically that the average MSE of two or more explanation methods will always be higher than the error of the averaged of those methods. Empirically, we test this for IROF with combinations of any two methods for ResNet101 and show the in fig. 5. For any two methods, the matrix shows the ratio between the aggregate method IROF and the average IROF of the aggregated methods. The aggregate IROF is always lower, confirming our theoretical . An important part of our method is the choice of explanation methods to be included in the aggregation. In our work we focus on backpropagation-based methods, since they tend to be computationally cheap. In our opinion this makes for a more realistic use case, not only for human but also for machine post processing. In contrast, locality based methods such as LIME or the method by require many forward passes, since their methods essentially "learn" what parts of the input are relevant. We included LIME in our experiments to have a not-backpropagation based method included. A.5.1 GENERAL We use SLIC for image segmentation due to availability and quick run time . Preliminary experiments with Quickshift showed similar . SLIC was chosen over Quickshift due to the quicker run time. The number of segments was set to 300 ad hoc. fig. 1 shows the same procedure with 100 segments for the sake of clarity. For AGG-Var, we add a constant to the denominator. We set this constant to 10 times the mean std, a value chosen empirically after trying values in the range of times the mean. Evaluations were run with a set random seed for reproducibility. Stddev were reported either for each individual or if they were non-significant in the caption to avoid cluttering the . Since our work does not include computationally heavy training, we did not record the exact computing infrastructure. The training for both models was equivalent. The architecture was as follows: We report p-values for evaluating with 50 images on ResNet101 in the manner described in section 4.2 in tabular form to provide a clear overview. We want to quantify whether an explanation method agrees with human judgement on which parts of an image should be important. While human annotation is expensive, there exists a benchmark for human evaluation introduced in. The benchmark includes ninety images of Figure 8: Example images from with human-annotated overlays. categories in the ImageNet Challenge (ten images were excluded due to the category not being in the ImageNet challenge) and provides annotations of relevant segments that ten human test objects found important. Example images are shown in fig. 8. While human evaluation is not a precise measure, we still expect some correlation between neural network and human judgement. To test the alignment, we calculate the cosine similarity, similarity(e j) = Since the images in this dataset are 224x224 pixel large, we only compute the cosine similarity for the network architectures where pretrained networks with this input size were available. We see that AGG-Mean and AGG-Var perform on-par with the best methods (SmoothGrad and GradCAM). While the aggregated methods perform better than the average explanation method, they do not surpass the best method. When we combine the two best-performing single methods, SmoothGrad and GradCAM, we surpass each individual method. We hypothesize that this is because the epistemic uncertainty is reduced by the aggregate.

We show in theory and in practice that combining multiple explanation methods for DNN benefits the explanation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: represent the article for finding a single-sentence tldr

Question: We present SOSELETO (SOurce SELEction for Target Optimization), a new method for exploiting a source dataset to solve a classification problem on a target dataset. SOSELETO is based on the following simple intuition: some source examples are more informative than others for the target problem. To capture this intuition, source samples are each given weights; these weights are solved for jointly with the source and target classification problems via a bilevel optimization scheme. The target therefore gets to choose the source samples which are most informative for its own classification task. Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting. SOSELETO may be applied to both classic transfer learning, as well as the problem of training on datasets with noisy labels; we show state of the art on both of these problems. Deep learning has made possible many remarkable successes, leading to state of the art algorithms in computer vision, speech and audio, and natural language processing. A key ingredient in this success has been the availability of large datasets. While such datasets are common in certain settings, in other scenarios this is not true. Examples of the latter include "specialist" scenarios, for instance a dataset which is entirely composed of different species of tree; and medical imaging, in which datasets on the order of hundreds to a thousand are common. A natural question is then how one may apply the techniques of deep learning within these relatively data-poor regimes. A standard approach involves the concept of transfer learning: one uses knowledge gleaned from the source (data-rich regime), and transfers it over to the target (data-poor regime). One of the most common versions of this approach involves a two-stage technique. In the first stage, a network is trained on the source classification task; in the second stage, this network is adapted to the target classification task. There are two variants for this second stage. In feature extraction (e.g.), only the parameters of the last layer (i.e. the classifier) are allowed to adapt to the target classification task; whereas in fine-tuning (e.g. BID12), the parameters of all of the network layers (i.e. both the features/representation and the classifier) are allowed to adapt. The idea is that by pre-training the network on the source data, a useful feature representation may be learned, which may then be recycled -either partially or completely -for the target regime. This two-stage approach has been quite popular, and works reasonably well on a variety of applications. Despite this success, we claim that the two-stage approach misses an essential insight: some source examples are more informative than others for the target classification problem. For example, if the source is a large set of natural images and the target consists exclusively of cars, then we might expect that source images of cars, trucks, and motorcycles might be more relevant for the target task than, say, spoons. However, this example is merely illustrative; in practice, the source and target datasets may have no overlapping classes at all. As a , we don't know a priori which source examples will be important. Thus, we propose to learn this source filtering as part of an end-to-end training process. The ing algorithm is SOSELETO: SOurce SELEction for Target Optimization. Each training sample in the source dataset is given a weight, corresponding to how important it is. The shared source/target representation is then optimized by means of a bilevel optimization. In the interior level, the source minimizes its classification loss with respect to the representation parameters, for fixed values of the sample weights. In the exterior level, the target minimizes its classification loss with respect to both the source sample weights and its own classification layer. The sample weights implicitly control the representation through the interior level. The target therefore gets to choose the source samples which are most informative for its own classification task. Furthermore, the bilevel nature of the optimization acts as a kind of regularization on the target, mitigating overfitting, as the target does not directly control the representation parameters. Finally, note that the entire processtraining of the shared representation, target classifier, and source weights -happens simultaneously. We pause here to note that the general philosophy behind SOSELETO is related to the literature on instance reweighting for domain adaptation, see for example BID32. However, there is a crucial difference between SOSELETO and this literature, which is related to the difference between domain adaptation and more general transfer learning. Domain adaptation is concerned with the situation in which there is either full overlap between the source and target label sets; or in some more recent work BID43, partial but significant overlap. Transfer learning, by contrast, refers to the more general situation in which there may be zero overlap between label sets, or possibly very minimal overlap. (For example, if the source consists of natural images and the target of medical images.) The instance reweighting literature is concerned with domain adaptation; the techniques are therefore relevant to the case in which source and target have the same labels. SOSELETO is quite different: it makes no such assumptions, and is therefore a more general approach which can be applied to both "pure" transfer learning, in which there is no overlap between source and target label sets, as well as domain adaptation. (Note also a further distinction with domain adaptation: the target is often -though not always -taken to be unlabelled in domain adaptation. This is not the case for our setting of transfer learning.)Above, we have illustrated how SOSELETO may be applied to the problem of transfer learning. However, the same algorithm can be applied to the problem of training with noisy labels. Concretely, we assume that there is a large noisy dataset, as well as a much smaller clean dataset; the latter can be constructed cheaply through careful hand-labelling, given its small size. Then if we take the source to be the large noisy dataset, and the target to the small clean dataset, SOSELETO can be applied to the problem. The algorithm will assign high weights to samples with correct labels and low weights to those with incorrect labels, thereby implicitly denoising the source, and allowing for an accurate classifier to be trained. The remainder of the paper is organized as follows. Section 2 presents related work. Section 3 presents the SOSELETO algorithm, deriving descent equations as well as convergence properties of the bilevel optimization. Section 4 presents of experiments on both transfer learning as well as training with noisy labels. Section 5 concludes. Transfer learning As described in Section 1, the most common techniques for transfer learning are feature extraction and fine-tuning, see for example and BID12, respectively. An older survey of transfer learning techniques may be found in BID25. Domain adaptation BID28 is concerned with transferring knowledge when the source and target classes are the same. Earlier techniques aligned the source and target via matching of feature space statistics; BID19; subsequent work used adversarial methods to improve the domain adaptation performance BID10; BID36;.In this paper, we are more interested in transfer learning where the source and target classes are different. A series of recent papers BID20; BID26 BID4 b) address domain adaptation that is closer to our setting. In particular, BID5 examines "partial transfer learning", the case in which there is partial overlap between source and target classes (particularly when the target classes are a subset of the source). This setting is also dealt with in BID3. BID11 examine the scenario where the source and target classes are completely different. Similar to SOSELETO, they propose selecting a portion of the source dataset. However, the selection is not performed in an end-to-end fashion, as in SOSELETO; rather, selection is performed prior to training, by finding source examples which are similar to the target dataset, where similarity is measured by using filter bank descriptors. Another recent work of interest is, which focuses on a slightly different scenario: the target consists of a very small number of labelled examples (i.e. the few-shot regime), but a very large number of unlabelled examples. Training is achieved via an adversarial loss to align the source and the target representations, and a special entropy-based loss for the unlabelled part of the data. Instance reweighting for domain adaptation is a well studied technique, demonstrated e.g. in Covariate Shift methods Shimodaira FORMULA6; BID31. In these works, the source and target label spaces are the same. We, however, allow for different -even entirely nonoverlapping -classes in the source and target. Crucially, we do not make assumptions on the similarity of the distributions nor do we explicitly optimize for it. The same distinction applies for the recent work of, and for the partial overlap assumption of BID43. In addition, these two works propose an unsupervised approach, whereas our proposed method is completely supervised. Covariate shift determines the weighting for an instance as the ratio of its probability of being in the training set and being in the prediction set. Consequently, the feature vectors are used in re-weighting, regardless of their labels. This renders covariate shift unsuitable for handling noisy labels. Our re-weighing scheme is instead gradient-based and as we show next performs well in this task. Learning with noisy labels Classification with noisy labels is a longstanding problem in the machine learning literature, see the review paper BID9 and the references therein. Within the realm of deep learning, it has been observed that with sufficiently large data, learning with label noise -without modification to the learning algorithms -actually leads to reasonably high accuracy BID14; BID34.The setting that is of greatest interest to us is when the large noisy dataset is accompanied by a small clean dataset. BID33 introduce an additional noise layer into the CNN which attempts to adapt the output to align with the noisy label distribution; the parameters of this layer are also learned. BID39 use a more general noise model, in which the clean label, noisy label, noise type, and image are jointly specified by a probabilistic graphical model. Both the clean label and the type of noise must be inferred given the image, in this case by two separate CNNs. consider the same setting, but with additional information in the form of a knowledge graph on labels. Other recent work on label noise includes BID27, which shows that adding many copies of an image with noisy labels to a clean dataset barely dents performance; , in which two separate networks are simultaneously trained, and a sample only contributes to the gradient descent step if there is disagreement between the networks (if there is agreement, that probably means the label is wrong); and BID8, which analyzes theoretically the situations in which CNNs are more and less resistant to noise. A pair of papers BID18; combine ideas of learning with label noise with instance reweighting. Bilevel optimization Bilevel optimization problems have a nested structure: the interior level (sometimes called the lower level) is a standard optimization problem; and the exterior level (sometimes called the upper level) is an optimization problem where the objective is a function of the optimal arguments from the interior level. A branch of mathematical programming, bilevel optimization has been extensively studied within this community BID6;. For recent developments, readers are referred to the review paper BID30. Bilevel optimization has been used in both machine learning, e.g. BID1 and computer vision, e.g. BID24. We have two datasets. The source set is the data-rich set, on which we can learn extensively. It is denoted by {(x DISPLAYFORM0, where as usual x s i is the i th source training image, and y s i is its corresponding label. The second dataset is the target set, which is data-poor; but it is this set which ultimately interests us. That is, the goal in the end is to learn a classifier on the target set, and the source set is only useful insofar as it helps in achieving this goal. The target set is denoted DISPLAYFORM1, and it is assumed that is much smaller than the source set, i.e. n t n s .Our goal is to exploit the source set to solve the target classification problem. The key insight is that not all source examples contribute equally useful information in regards to the target problem. For example, suppose that the source set consists of a broad collection of natural images; whereas the target set consists exclusively of various breeds of dog. We would assume that any images of dogs in the source set would help in the target classification task; images of wolves might also help, as might cats. Further afield it might be possible that objects with similar textures as dog fur might be useful, such as rugs. On the flip side, it is probably less likely that images of airplanes and beaches will be relevant (though not impossible). However, the idea is not to come with any preconceived notions (semantic or otherwise) as to which source images will help; rather, the goal is to let the algorithm choose the relevant source images, in an end-to-end fashion. We assume that the source and target classifier networks have the same architecture, but different network parameters. In particular, the architecture is given by DISPLAYFORM2 where φ is last layer, or possibly last few layers, and θ constitutes all of the remaining layers. We will refer to φ colloquially as the "classifier", and to θ as the "features" or "representation". (This is consistent with the usage in related papers, see for example .) Now, the source and target will share features, but not classifiers; that is, the source network will be given by F (x; θ, φ s), whereas the target network will be F (x; θ, φ t). The features θ are shared between the two, and this is what allows for transfer learning. The weighted source loss is given by DISPLAYFORM3 where α j ∈ is a weight assigned to each source training example; and (·, ·) is a per example classification loss, in this case cross-entropy. The use of the weights α j will allow us to decide which source images are most relevant for the target classification task. The target loss is standard: DISPLAYFORM4 As noted in Section 1, this formulation allows us to address both the transfer learning problem as well as learning with label noise. In the former case, the source and target may have non-overlapping label spaces; high weights will indicate which source examples have relevant knowledge for the target classification task. In the latter case, the source is the noisy dataset, the target is the clean dataset, and they share a classifier (i.e. φ t = φ s) as well as a label space; high weights will indicate which source examples do not have label noise, and are therefore reliable. In either case, the target is much smaller than the source. The question now becomes: how can we combine the source and target losses into a single optimization problem? A simple idea is to create a weighted sum of source and target losses. Unfortunately, issues are likely to arise regardless of the weight chosen. If the target is weighted equally to the source, then overfitting may likely given the small size of the target. On the other hand, if the weights are proportional to the size of the two sets, then the source will simply drown out the target. A more promising idea is to use bilevel optimization. Specifically, in the interior level we find the optimal features and source classifier as a function of the weights α, by minimizing the source loss: DISPLAYFORM5 In the exterior level, we minimize the target loss, but only through access to the source weights; that is, we solve: min DISPLAYFORM6 Why might we expect this bilevel formulation to succeed? The key is that the target only has access to the features in an indirect manner, by controlling which source examples are included in the source classification problem. Thus, the target can influence the features chosen, but only in this roundabout way. This serves as an extra form of regularization, mitigating overfitting, which is the main threat when dealing with a small set such as the target. Implementing the bilevel optimization is rendered somewhat challenging due to the need to solve the optimization problem in the interior level. Note that this optimization problem must be solved at every point in time; thus, if we choose to solve the optimization for the exterior level via gradient descent, we will need to solve the interior level optimization at each iteration of the gradient descent. This is clearly inefficient. Furthermore, it is counter to the standard deep learning practice of taking small steps which improve the loss. Thus, we instead propose the following procedure. At a given iteration, we will take a gradient descent step for the interior level problem: DISPLAYFORM7 where m is the iteration number; λ p is the learning rate (where the subscript p stands for "parameters", to distinguish it from a second learning rate for α, to appear shortly); and Q(θ, φ s) is a matrix whose j th column is given by DISPLAYFORM8 Thus, Equation leads to an improvement in the features θ, for a fixed set of source weights α. Note that there will be an identical descent equation for the classifier φ s, which we omit for clarity. Given this iterative version of the interior level of the bilevel optimization, we may now turn to the exterior level. Plugging Equation into Equation FORMULA6 gives the following problem: DISPLAYFORM9 DISPLAYFORM10 where we have suppressed Q's arguments for readability. We can then take a gradient descent step of this equation, yielding: DISPLAYFORM11 where in the final line, we have made use of the fact that λ p is small. Of course, there will also be a descent equation for the classifier φ t. The ing update scheme is quite intuitive: source example weights are update according to how well they align with the target aggregated gradient. We have not yet dealt with the weight constraint. That is, we would like to explicitly require that each α j ∈. We may achieve this by requiring α j = σ(β j) where the new variable β j ∈ R, and σ: R → is a sigmoid-type function. As shown in Appendix A, for a particular piecewise linear sigmoid function, replacing the Update Equation with a corresponding update equation for β is equivalent to modifying Equation to read DISPLAYFORM12 where CLIP clips the values below 0 to be 0; and above 1 to be 1. FORMULA7 and FORMULA12, along with the descent equations for the source and target classifiers φ s and φ t. As usual, the whole operation is done on a mini-batch basis, rather than using the entire set; note that if processing is done in parallel, then source minibatches are taken to be non-overlapping, so as to avoid conflicts in the weight updates. SOSELETO is summarized in Algorithm 1. Note that the target derivatives ∂L t /∂θ and ∂L t /∂φ t are evaluated over a target mini-batch; we suppress this for clarity. In terms of time-complexity, we note that each iteration requires both a source batch and a target batch; assuming identical batch sizes, this means that SOSELETO requires about twice the time as the ordinary source classification problem. Regarding space-complexity, in addition to the ordinary network parameters we need to store the source weights α. Thus, the additional relative spacecomplexity required is the ratio of the source dataset size to the number of network parameters. This is obviously problem and architecture dependent; a typical number might be given by taking the source dataset to be Imagenet ILSVRC-2012 (size 1.2M) and the architecture to be ResNeXt-101 BID40 (size 44.3M parameters), yielding a relative space increase of about 3%.Convergence properties SOSELETO is only an approximation to the solution of a bilevel optimization problem. As a , it is not entirely clear whether it will even converge. In Appendix B, we demonstrate a set of sufficient conditions for SOSELETO to converge to a local minimum of the target loss L t. We briefly discuss some implementation details. In all experiments, we use the SGD optimizer without learning rate decay, and we use λ α = 1. We initialize the α-values to be 1, and in practice clip them to be in the slightly expanded range [0, 1.1]; this allows more relevant source points some room to grow. Other settings are experiment specific, and are discussed in the relevant sections. To illustrate how SOSELETO works on the problem of learning with noisy labels, we begin with a synthetic experiment, see FIG0. The setting is straightforward: the source dataset consists of 500 points which lie in R 2. There are two labels / classes, and the ideal separator between the classes is the y-axis. However, of the 500 points, 100 are corrupted: that is, they lie on the wrong side of the separator. This is shown in FIG0, in which one class is shown as white triangles and the second as black pluses. The target dataset is a set of 50 points, which are "clean", in the sense that they lie on the correct sides of the separator. (For the sake of simplicity, the target set is not illustrated.) SOSELETO is run for 100 epochs. In FIG0 and 1(c), we choose a threshold of 0.1 on the weights α, and colour the points accordingly. In particular, in FIG0 (b) the clean (i.e. correctly labelled) instances which are above the threshold are labelled in green, while those below the threshold are labelled in red; as can be seen, all of the clean points lie above the threshold for this choice of threshold, meaning that SOSELETO has correctly identified all of the clean points. In FIG0 (c), the noisy (i.e. incorrectly labelled) instances which are below the threshold are labelled in green; and those above the threshold are labelled in red. In this case, SOSELETO correctly identifies most of these noisy labels by assigning them small weights (below 0.1); in fact, 92 out of 100 points are assigned such small weights. The remaining 8 points, those shown in red, are all near the separator, and it is therefore not very surprising that SOSELETO mislabels them. All told, using this particular threshold the algorithm correctly accounts for 492 out of 500 points, i.e. 98.4%. FIG0. In FIG0 (e), a plot is shown of mean weight vs. training epoch for clean instances and noisy instances; the width of each plot is the 95% confidence interval of the weights of that type. All weights are initialized at 0.5; after 100 epochs, the clean instances have a mean weight of about 0.8, whereas the noisy instances have a mean weight of about 0.05. The evolution is exactly as one would expect. FIG0 (e) examines the role of the threshold, chose as 0.1 in the above discussion; although 0.1 is a good choice in this case, the good behaviour is fairly robust to choices in the range of 0.1 to 0.4. We now turn to a real-world setting of the problem of learning with label noise. We use a noisy version of CIFAR-10 , following the settings used in BID33; BID39. In particular, an overall noise level is selected. Based on this, a label confusion matrix is chosen such that the diagonal entries of the matrix are equal to one minus the noise level, and the off-diagonals are chosen randomly (while maintaining the matrix's stochasticity). Noisy labels are then sampled according to this confusion matrix. We run experiments for various overall noise levels. The target consists of a small clean dataset. CIFAR-10's train set consists of 50K images; of this 50K, both BID33; BID39 set aside 10K clean examples for pre-training, a necessary step in both of these algorithms. In contrast, we use a smaller clean dataset of half the size, i.e. 5K examples while the rest of the 45K samples are noisy. We compare our to the two state of the art methods BID33; BID39, as they both address the same setting as we do -the large noisy dataset is accompanied by a small clean dataset, with no extra side-information available. In addition, we compare with the baseline of simply training on the noisy labels without modification. In all cases, Caffes CIFAR-10 Quick cif architecture has been used. For SOSELETO, we use the following settings: λ p = 10 −4, the target batch-size is 32, and the source batch-size is 256. We use a larger source batch-size to enable more α-values to be affected quickly. Results are shown in TAB1 for three different overall noise levels, 30%, 40%, and 50%. Performance is reported for CIFAR-10's test set, which is of size 10K. (Note that the competitors' performance numbers are taken from BID39 .) SOSELETO achieves state of the art on all three noise levels, with considerably better performance than both BID33 and BID39: between 2.6% to 3.2% absolute improvement. Furthermore, it does so in each case with only half of the clean samples used in BID33 BID39.We perform further analysis by examining the α-values that SOSELETO chooses on convergence, see Figure 4.2. To visualize the , we imagine thresholding the training samples in the source set on the basis of their α-values; we only keep those samples with α greater than a given threshold. By increasing the threshold, we both reduce the total number of samples available, as well as change the effective noise level, which is the fraction of remaining samples which have incorrect labels. We may therefore plot these two quantities against each other, as shown in Figure 4.2; we show three plots, one for each noise level. Looking at these plots, we see for example that for the 30% noise level, if we take the half of the training samples with the highest α-values, we are left with only about 4% which have incorrect labels. We can therefore see that SOSELETO has effectively filtered out the incorrect labels in this instance. For the 40% and 50% noise levels, the corresponding numbers are about 10% and 20% incorrect labels; while not as effective in the 30% noise level, SOSELETO is still operating as designed. Further evidence for this is provided by the large slopes of all three curves on the righthand side of the graph. We now examine the performance of SOSELETO on a transfer learning task. In order to provide a challenging setting, we choose to (a) use source and target sets with disjoint label sets, and (b) use a very small target set. In particular, the source dataset is chosen to the subset of Google Street View House Numbers (SVHN) BID23 corresponding to digits 0-4. SVHN's train set is of size 73,257 images, with about half of those belonging to the digits 0-4. The target dataset is a very small subset of MNIST BID16 corresponding to digits 5-9. While MNIST's train set is of size 60K, with 30K corresponding to digits 5-9, we use very small subsets: either 20 or 25 images, with equal numbers sampled from each class (4 and 5, respectively). Thus, as mentioned, there is no overlap between source and target classes, making it a true transfer learning (rather than domain adaptation) problem; and the small target set size adds further challenge. Furthermore, this task has already been examined in.We compare our with the following techniques. Target only, which indicates training on just the target set; standard fine-tuning; Matching Nets BID38, a few-shot technique which is relevant given the small target size; fine-tuned Matching Nets, in which the previous is then fine-tuned on the target set; and two variants of the Label Efficient Learning technique -one which includes fine-tuning plus a domain adversarial loss, and the other the full technique presented in. Note that besides the target only and fine-tuning approaches, all other approaches depend on unlabelled target data. Specifically, they use all of the remaining MNIST 5-9 examples -about 30,000 -in order to aid in transfer learning. SOSELETO, by contrast, does not make use of any of this data. For each of the above methods, the simple LeNet architecture BID16 was used. For SOSELETO, we use the following settings: λ p = 10 −2, the source batch-size is 32, and the target batch-size is 10 (it is chosen to be small since the target itself is very small). Additionally, the SVHN images were resized to 28 × 28, to match the MNIST size. The performance of the various methods is shown in TAB2, and is reported for MNIST's test set which is of size 10K. We have divided TAB2 into two parts: those techniques which use the 30K examples of unlabelled data, and those which do not. SOSELETO has superior performance to all of the techniques which do not use unlabelled data. Furthermore, SOSELETO has superior performance to all of the techniques which do use unlabelled data, except the Label Efficient technique. It is noteworthy in particular that SOSELETO outperforms the few-shot techniques, despite not being designed to deal with such small amounts of data. In Appendix C we further analyze which SVHN instances are considered more useful than others by SOSELETO, by transfering all of SVHN classes to MNSIT 5-9. Two-stage SOSELETO Finally, we note that although SOSELETO is not designed to use unlabelled data, one may do so using the following two-stage procedure. Stage 1: run SOSELETO as described above. Stage 2: use the learned SOSELETO classifier to classify the unlabelled data. This will now constitute a dataset with noisy labels, and SOSELETO can now be run in the mode of training with label noise, where the noisily labelled unsupervised data is now the source, and the target remains the same small clean set. In the case of n t = 25, this procedure elevates the accuracy to above 92%. We have presented SOSELETO, a technique for exploiting a source dataset to learn a target classification task. This exploitation takes the form of joint training through bilevel optimization, in which the source loss is weighted by sample, and is optimized with respect to the network parameters; while the target loss is optimized with respect to these weights and its own classifier. We have derived an efficient algorithm for performing this bilevel optimization, through joint descent in the network parameters and the source weights, and have analyzed the algorithm's convergence properties. We have empirically shown the effectiveness of the algorithm on both learning with label noise, as well as transfer learning problems. An interesting direction for future research involves incorporating an additional domain alignment term into SOSELETO, in the case where the source and target dataset have overlapping labels. We note that SOSELETO is architecture-agnostic, and thus may be easily deployed. Furthermore, although we have focused on classification tasks, the technique is general and may be applied to other learning tasks within computer vision; this is an important direction for future research. Recall that our goal is to explicitly require that α j ∈. We may achieve this by requiring DISPLAYFORM0 where the new variable β j ∈ R, and σ(·) is a kind of piecewise linear sigmoid function. Now we will wish to replace the Update Equation, the update for α, with a corresponding update equation for β. This is straightforward. Define the Jacobian ∂α/∂β by ∂α ∂β ij = ∂α i ∂β jThen we modify Equation to read DISPLAYFORM1 The Jacobian is easy to compute analytically: where CLIP clips the values below 0 to be 0; and above 1 to be 1. DISPLAYFORM2 SOWETO is only an approximation to the solution of a bilevel optimization problem. As a , it is not entirely clear whether it will even converge. In this section, we demonstrate a set of sufficient conditions for SOWETO to converge to a local minimum of the target loss L t.To this end, let us examine the change in the target loss from iteration m to m + 1: DISPLAYFORM0 Now, we can use the evolution of the weights α. Specifically, we substitute Equation FORMULA11

Learning with limited training data by exploiting "helpful" instances from a rich data source.

represent the tldr for finding the corresponding scientific article

For example, This paper proposes a new method for neural network learning in online bandit settings by marginalizing over the last layer should be similar to We propose a new method for training neural networks online in a bandit setting. Similar to prior work, we model the uncertainty only in the last layer of the network, treating the rest of the network as a feature extractor. This allows us to successfully balance between exploration and exploitation due to the efficient, closed-form uncertainty estimates available for linear models. To train the rest of the network, we take advantage of the posterior we have over the last layer, optimizing over all values in the last layer distribution weighted by probability. We derive a closed form, differential approximation to this objective and show empirically over various models and datasets that training the rest of the network in this fashion leads to both better online and offline performance when compared to other methods. Applying machine learning models to real world applications almost always involves deploying systems in dynamic, non-stationary environments. This dilemma requires models to be constantly re-updated with new data in order to maintain a similar model performance across time. Of course, doing this usually requires the new data to be relabeled, which can be expensive or in some cases, impossible. In many situations, this new labeled data can be cheaply acquired by utilizing feedback from the user, where the feedback/reward indicates the quality of the action taken by the model for the given input. Since the inputs are assumed independent, this task can be framed in the contextual bandit setting. Learning in this setting requires a balance between exploring uncertain actions (where we risk performing sub optimal actions) and exploiting actions the model is confident will lead to high reward (where we risk missing out on discovering better actions).Methods based on Thompson sampling (TS) or Upper Confidence Bounds (UCB) provide theoretically BID1 BID0 and empirically established ways BID12 BID3 for balancing exploration/exploitation in this setting. Unfortunately, both methods require estimation of model uncertainty. While this can be done easily for most linear models, it is a difficult and open problem for large neural network models underlying many modern machine learning systems. An empirical study by BID17 shows that having good uncertainty estimates is vital for neural networks learning in a bandit setting. Closed formed uncertainty estimations (and online update formulas) are available for many linear models. Since the last layer of many neural networks are usually (generalized) linear models, a straightforward way for learning neural networks in a bandit setting is to estimate the uncertainty (as a distribution over weights) on the last layer only, holding the previous layers fixed as feature functions which provide inputs to the linear model. This method (and variants thereof) has been proposed in bandit settings BID17 BID13 as well as other related settings (; BID16 BID11 and has been shown to work surprisingly well considering its simplicity. This style of methods, which we refer to as Bayesian last layer or BLL methods, also has the advantage of being both relatively model-agnostic and scalable to large models. Of course, BLL methods come with the tacit assumption that the feature functions defined by the rest of the network output good (linearly separable) representations of our inputs. This means that, unless the input data distribution is relatively static, the rest of the network will need to be updated in regular intervals to maintain low regret. In order to maintain low regret, the retraining objective must: 1) allow new data to be incorporated quickly into the learned model, and 2) prevent previously learned information from being quickly forgotten. Previous papers retrain BLL methods simply by sampling minibatches from the entire pool of previously seen data and maximizing log-likelihood over these minibatches, which fails to meet the first criteria above. In this paper we present a new retraining objective for BLL methods meeting both requirements. We avoid retraining the last layer with the entire network (throwing out the uncertainty information we learned about the last layer) or retraining with the last layer fixed (fixing the last layer to the mean of its distribution). Instead, we utilize the uncertainty information gathered about the last layer, and optimize the expected log-likelihood of both new and old data, marginalizing 1 over the entire distribution we have on the last layer. This gives a more robust model that performs relatively well over all likely values of the last layer. While this objective cannot be exactly computed, we derive a closed form, differentiable, approximation. We show that this approximation meets both criteria above, with a likelihood term to maximize that depends only on the new data (meeting the first point), and a quadratic regularization term that is computed only with previously seen data (meeting the second point). We show empirically that this method improves regret on the most difficult bandit tasks studied in BID17. We additionally test the method on a large state-of-the-art recurrent model, creating a bandit task out of a paraphrasing dataset. Finally, we test the method on convolutional models, constructing a bandit task from a benchmark image classification dataset. We show that our method is fast to adapt to new data without quickly forgetting previous information. Contextual bandits are a well researched class of sequential decision problems (; BID10, of which, many variants exist. In this paper, we mainly consider the multiclass contextual bandit problem studied in BID7 . The problem takes place over T online rounds. On round t, the learner receives a context x t, predicts a class label y t, and receives binary reward r t indicating whether the chosen label is correct. No other information is received about the other classes not picked. In our setting, we assume each class c (the arms of the bandit) has associated with it a vector z c and that the probability of a class label is modeled by: DISPLAYFORM0, where σ is the logistic function and f θ is a feature function parameterized by θ. In our case, f θ defines a neural network, while z c can be seen as the last layer of the network 2.Our goal is to get low regret, R = T i r * i − T i r i, where r * i is the optimal reward at step i. The key to getting low regret is employing a policy for balancing exploration and exploitation. If we capture the uncertainty in each z c at time t by modeling its posterior distribution over previous data D t−1 as a multivariate Gaussian, z c ∼ p(z c |D t−1), then we can easily deploy sound exploration strategies such as Thompson sampling or UCB. If we hold f θ fixed, then we can easily model this distribution by doing an online Bayesian regression on the outputs of f θ, which gives us closed form formulas for updating the posterior over the last layer (specifically, its mean and covariance) given a single datapoint.3 When f θ is a neural network, then this is an instance of a BLL method. BLL methods have been shown to be an effective, model agnostic, and scalable way to deal with exploration problems involving neural networks. Previous work has found them to be a pragmatic method for obtaining approximate uncertainty estimates for exploration BID13 BID17 BID16 BID2 and as proxies for Gaussian processes in both Bayesian Optimization problems and general regression tasks BID11.If f θ is fixed, then z c can be updated efficiently in an online manner. An unanswered question still remains however: how does one actually update and learn f θ? If you don't care about achieving low regret (ie you only care about offline performance), then the answer is easy; just gather your data, train f θ offline, possibly with off-policy learning methods (; BID6, and learn the Bayesian regression post-hoc. Of course, if you are concerned about online performance (regret) then this is not a viable option. A training method for f θ must take care of two things: when do we update the feature functions and what do we update them with? A reasonable answer to the former question is to update on a fixed schedule (every T rounds). In this paper, we focus on answering the latter questions of which there are two obvious solutions, each with a corresponding problem: Sample minibatches only from recent data. Problem: We may overfit on this data and forget old information. Sample minibatches uniformly from the set of all collected data (both recent and old). Problem:We have lost the ability to adapt quickly to new data. If the input distribution suddenly shifts, we will likely have to wait many iterations before newer data becomes common enough in our minibatch samples, all while our regret increases. One thing to consider is that when it comes time to train our feature functions, we have access to a distribution over our last layer. If, for example, our distribution has suddenly shifted, then the last layer distribution should have more variance, ideally placing density on last layers that do well on older data, as well as those that fit well to the new data. If the distribution remains the same, then the variance should be low and density should be placed on relatively few values. Intuitively, we can get the best of both worlds (ability to adapt or retain information when needed) by optimizing over all values of the last layer weighted by their probability. In the next section, we derive a local approximation to this objective that shows this indeed is the case. Let D t and D t−1 be the most recent set of data collected online, and the set of all previously collected data, respectively. Additionally, assume a zero mean Gaussian prior over the last layer, p(z|θ) that is constant with respect to θ. Recall that during online training we fix the parameters θ = θ t−1, and model the distribution Q = p(z c |D t, D t−1, θ t−1). We want a value of θ such that both our data and the last layer values drawn from Q are likely. Thus our objective is to maximize: DISPLAYFORM0 We can write the marginal likelihood as a convolution between a logistic function and a Gaussian (based on our assumption of zero mean Gaussian prior p(z|θ)), which can be approximated in closed form as per MacKay FORMULA1: DISPLAYFORM1 Where µ is the mean of p(z|θ). Since we have a zero mean prior, the above term evaluates to σ whose value is a constant 1 2. Using this , we can rewrite equation FORMULA1 as: DISPLAYFORM2 Where c is a constant entropy term which can be ignored. For the first expectation term, we can use a second order Taylor expansion around E z∼Q [log p(D t |z, θ)] to get a closed form approximation 4: DISPLAYFORM3 This approximation was used in and shown to work well empirically. The expectations in equation FORMULA4 The KL term in equation FORMULA3 can also be approximated locally with a second Taylor expansion around the current value of θ = θ t−1. Let K(θ) = KL(Q||p(z|D t−1, θ)). Then, the second order Taylor expansion around θ = θ t−1 is: DISPLAYFORM4 Utilizing properties of KL divergence, as well as equation FORMULA2, it can be derived 5 that K (θ t−1) will evaluate to 0, and K (θ t−1) will evaluate to βF P, where β = DISPLAYFORM5 and F P is the Fisher Information Matrix of P = p(z|D t−1, θ t−1). Getting rid of constants, we can write the local KL approximation (when θ is close to θ t−1) as: DISPLAYFORM6 The term β defines the ratio between the expected data likelihood given the last layer z distributed as z ∼ p(z|D t−1, θ t−1) and the expected data likelihood given the last layer is distributed under the prior p(z|θ). This indicates that the better our previous values of θ t−1 and z explain the data, the more we should regularize when incorporating new data (i.e raising the value of β). In practice, these values may be computed or approximated, however for efficiency, we treat β as a hyperparameter, and linearly increase it throughout the learning process. Our final objective to optimize is thus: DISPLAYFORM7 Notice that the old data is now only used to calculate the Fisher information matrix F P and is not actually involved in the optimization. Thus, the optimization (at least temporarily) over all our data can be done by simply drawing minibatches from the new data only, while the old data is only used to calculate the regularization term. The practical benefit of this is that the regularization term can be easily computed in parallel while doing the online Bayesian regression and collecting new data. The quadratic regularization term shares similarities to objective functions in continual learning which aim to prevent catastrophic forgetting BID8 ). Combining the online learning and the network retraining stages described in the previous section gives us the general form of the iterative algorithm we study in this paper. The algorithm alternates between two stages:Online Phase: As input, take in a set of data D t−1, the posteriors (one for each arm) of the last layer conditioned on previous data p(z|D t−1) as well as a fixed value of f θ. This phase takes place over a series of T online rounds. In every round, the learner receives a context x i, and uses the posteriors over the last layer to decide which arm to pull (via Thompson sampling or UCB). The learner receives feedback y i upon pulling the arm c, and updates the posterior over z c with it. After T rounds, the learner outputs the updated posteriors over z, and the data collected this phase, D t.Offline/Retraining Phase: As input, take in D t, D t−1, and the posteriors over z. Retrain f θ using method described in Section 3. Set D t−1 = D t ∪D t−1. Recompute the posteriors over z conditioned on D t−1 using the new value of f θ. Output D t−1, f θ, and the updated posteriors p(z|D t−1).The marginalization method described in Section 3 is one type of retraining method. We compare it against two other methods in the next section and present for various experiments. We evaluate our technique across a diverse set of datasets and underlying models. As an initial sanity check, we first evaluate our method on the three most difficult (but low dimensional) bandit tasks analyzed in BID17. We next look at two higher dimensional problems and models; one being a Natural Language Processing (NLP) task using a state-of-the-art recurrent model and the other being a vision task using a convolutional model. In particular, we look at the degree at which each method can achieve both good online performance (regret) and good offline performance (offline test set accuracy), even in the face of large shifts in the data distribution. We provide additional details on hyperparameters, experimental setup, and dataset information in Appendix 7.4. All experiments are run 5 times, and reported with the mean and standard error. We evaluate all the datasets against the baseline presented in BID17, as well as a variant of our proposed method. Bandit feedback. In this setting the models are trained using bandit feedback as the label:Marginalize. This is our method of marginalizing over all values of the last layer for the neural network training. Minibatches are sampled from the new data only and the regularization term is computed from the old data. Sample New. This baseline creates minibatches using only the newly collected data, optimizing the likelihood of the new data only. It is equivalent to our method with a regularization constant of zero. As mentioned in Section 2.1, this method is good at adapting to new data but has a drawback of forgetting the old information. Sample All. This is the retraining method presented in BID17. In this method, a set number minibatches are created by uniformly sampling from all collected data (both old and new). SGD gradient updates are then performed using these batches. This method is slow to adapt but retains older information (refer Section 2.1).Full feedback. In this setting models are trained using all the labels for the datasets:Batch Train. When evaluating the offline accuracy, we also give the for a model that has been trained on the shuffled data in batch mode, with all the labels for the dataset (i.e. full feedback). This measures how well we could do given we had access to all the labels (instead of just the bandit feedback), and trained in a normal offline setting. Surprisingly, as we see in some cases, training online with marginalization sometimes performs comparable to training offline. We first confirm that our method gives good online performance on simpler, but previously studied, problems in BID17.We present on the three hardest bandit tasks analyzed in BID17, the Census, Jester, and Adult dataset. The bandit problems for these datasets are defined as in previous work:Census and Adult. Both these datasets are used for multiclass classification problem. Census dataset has 9 classes whereas Adult dataset consists of 14 classes. For both datasets the bandit problem is created as follows: for each class we assign an arm, and each arm is associated with a logistic regression (parametrized by a vector) that takes a context as input and returns the expected reward (0 or 1) for selecting the arm. In the online round we receive a context (feature vector) and pick an arm according to some policy (like UCB), and receive a reward. Only the picked arm is updated in each round. Jester BID5 This dataset consists of jokes with their user rating. For the bandit problem, the model receives a context representing a user along with 8 jokes out of which it is required to make recommendation of 1 joke. In this setting each joke is defined as a bandit arm. The problem here is similar to above with the exception that each arm is associated with a linear regression and outputs the predicted user rating for the selected joke. The reward returned is the actual user rating for the selected joke. As previously done in BID17, we use a two layer MLP as the underlying model, using the same configuration across all methods. For Marginalize and Sample New, we perform the retraining after 1000 rounds. For Sample All we update after 100 rounds just like BID17. In Table 1 we report the average cumulative regret as well as the cumulative regret relative to a policy that selects arms uniformly at random. We report the for both Thompson Sampling (TS) and for UCB policies. Results are similar for either UCB and TS which shows that policies does not influence performance of the training mechanisms. On most of the tasks both Marginalize (our method) and Sample New outperforms Sample All (method used in BID17) in terms of cumulative regret. Both Marginalize and Sample New techniques are very similar in performance for the three datasets. All the three datasets used in this experiment are low dimensional, static, and relatively easy to learn, hence there is not much history to retain for Sample New technique. In the next section we will present on larger datasets and also evaluate where we will show that our method performs better than Sample New. Next we evaluate our method with a bigger and more complex underlying model on the NLP domain. We selected Bilateral Multi-Perspective Matching (BiMPM) , a recurrent model that performs well on several sentence matching tasks, including the paraphrase identification task, to evaluate our method. The goal of the paraphrase identification task is to determine whether a sentence is a paraphrase of another sentence, i.e., whether they convey the same meaning. To evaluate whether our algorithm is robust to shifts in data distribution we combined two different paraphrase identification datasets: i) The Quora Question Pairs dataset (Quora), 6 which contains 400,000 question pairs from the QA website Quora.com, and ii) The MSR Paraphrase Corpus (MSR) BID4, which contains 5,800 pairs of sentences extracted from news articles. To create an online training dataset we concatenate the MSR training set to a sample of 10,000 examples from the Quora training dataset 7.We run the online algorithms on this dataset to report the regret values, while we report the offline performance on the MSR and Quora test sets. We use UCB as our search strategy, as it performs similarly to Thompson sampling and runs much faster in our implementation. We analyze the following two bandit tasks:Multiclass. Like the previous datasets, we create a bandit problem by treating each class as an arm parameterized by a vector and the contexts as the individual data instances. A reward 1 is awarded for identifying correctly if the two sentences in the pair are paraphrase. For each method, we perform an offline retraining after 1,000 online rounds. Pool. Like the multiclass task, the pool based task occurs over a series of rounds. On each round, the model receives a pool of k(=3) instances, and must select one of them for the user. After that the model receives a reward based on its selection. The goal of the model is to learn a scoring function that predicts the expected reward for selecting a certain instance, while at the same time trying to keep regret low. This setting can be seen as an instance of the bandit problem formulation described in . In our case, our instances are candidate paraphrase pairs, where the model gets a reward of 1 for returning a valid paraphrase pair, and 0 otherwise. We use the same implementation and hyperparameters for BiMPM as in . For Marginalize and Sample All, we perform the retraining every 500 rounds. Sample New performed poorly offline on this setting and is thus updated every 1,000 rounds. In TAB2 we show that our method Marginalize outperforms both Sample All and Sample New techniques for both multiclass and pool based tasks. Sample All and Sample New have comparable cumulative regret. Sample New has worse offline accuracy on Quora dataset (because it forgets old information), while it has better offline accuracy on MSR (because it is able to adapt quicker). For Batch train, both multiclass and pool based tasks are same-a binary classification problem. Batch train performs only slightly better than our method in terms of offline accuracy, where Batch train gets full feedback, while our method only gets partial (bandit) feedback. FIG1 further shows that when the data distribution changes (switching form Quora to MSR in the pool based task) Marginalize and Sample New are able to adapt much faster than Sample All. Overall Marginalize achieved a lower regret as well as higher offline accuracy for both the bandit settings. We additionally evaluate our method using a convolutional neural network, which is a common network architecture for computer vision applications. Table 3: Image classification Bandit Task on CIFAR-10 dataset. We use CIFAR-10 dataset BID9 ) for this experiment. It is a commonly used dataset for image classification task consisting of 60,000 images from 10 different classes. Similar to the Quora/MSR tasks, we simulate a domain shift by concatenating together two datasets. In this case we create two data sets from CIFAR-10 by partitioning the dataset into images depicting animals (6 classes) and images depicting transportation (4 labels). As above, we analyze two bandit tasks:Multiclass. We define the multiclass bandit task similarly as above, for each of the 10 classes in CIFAR-10, we assign one arm. At each round, the model receives an image, guesses a class, and receives feedback (0 or 1) for this class only. The task is considerably more difficult than the multiclass paraphrase bandit due to the number of classes. We use 1,000 rounds for the retraining frequency for all methods. Pool. We also define a pool based bandit, similar to the pool based paraphrase bandit, with pool size k = 5. In this case we turn CIFAR-10 info a binary classification task. We select the two most difficult classes to classify (airplanes and birds, according to the confusion matrix of our base CNN model) in CIFAR-10, and denote these as the positive class. Like the previous pool task, the learner receives a pool of images and must select one. A reward of 1 is given for selecting an image belonging to the positive class, 0 otherwise. As done in the previous pool task, the data is sorted as to simulate a change in domain. We use a standard convolutional neural network architecture for this task, detailed in Appendix 7.5. We use 500 rounds for the retraining frequency for all methods. In Table 3 we present for the image classification bandit task, using average cumulative regret and offline accuracy as evaluation metrics. Again, Sample New performs better than Sample All for cumulative regret but under-performs in the offline setting. As expected, our method performs well for both cumulative regret and offline setting. For the multiclass task, our method performs significantly lower than batch train. This is not too surprising, for two reasons: i) Training a CNN architecture takes many more epochs over the data to converge (∼ 20 in our case) which is not achieved in a bandit setting; ii) CIFAR-10 has 10 classes, each defining an arm and in our setting; the bandit algorithms only gets feedback for one class in each round, compared to the full feedback received in batch train. Effectively, the number of labels per class in cut by a factor of 10. This is not as much an issue in the pool task, where we can see the between batch train and the bandit algorithms are comparable. In this paper we proposed a new method for training neural networks in a bandit setting. We tackle the problem of exploration-exploitation by estimating uncertainty only in the last layer, allowing the method to scale to large state-of-the-art models. We take advantage of having a posterior over the last layer weights by optimizing the rest of the network over all values of the last layer. We show that method outperforms other methods across a diverse set of underlying models, especially in online tasks where the distribution shifts rapidly. We leave it as future work to investigate more sophisticated methods for determining when to retrain the network, how to set the weight (β) of the regularization term in a more automatic way, and its possible connections to methods used for continual learning. We utilize the same notation here as in Section 3. First, we show how to arrive at equation from the main objective, equation FORMULA1. DISPLAYFORM0 By marginalizing over our zero mean Gaussian prior and using equation FORMULA2, it follows that the last expectation is approximately a constant, and can be removed. The second expectation is equal to the negative cross entropy between Q = p(z|D t, D t−1, θ t−1) and P θ = p(z|D t−1, θ). Using the fact that KL(Q||P θ) = CE(Q||P θ) − H(Q), we can replace the negative cross entropy with −KL(Q||P θ) − H(Q), where H(Q) is the entropy of Q which is constant and can be ignored, yielding equation.We next derive the KL term. As mentioned, we approximate the KL term locally around θ t−1 with a second order Taylor expansion. Again, let K(θ) = KL(p(z|D t, D t−1, θ t−1)||p(z|D t−1, θ)). The second order Taylor expansion around θ = θ t−1 is: DISPLAYFORM1 We can rewrite K (θ) with respect to θ as: DISPLAYFORM2 This can also be done for K. Let ∇ i indicate either the gradient (i = 1) or the Hessian (i = 2). Then we can rewrite the above expectation with respect to the distribution P = p(z|D t−1, θ t−1) instead: DISPLAYFORM3 Using the fact that p(z|D t, D t−1, θ t−1) = p(D t |θ t−1) −1 p(D t |z, θ t−1)p(z|D t−1, θ t−1), we can rewrite the above as: DISPLAYFORM4 the UCB is straightforward; if Σ is the covariance of our posterior over z, and x is the context, then with probability at least 1 − δ, the term (1 + ln(2/δ)/2) f θ (x) T Σf θ (x) is an upper confidence bound. The term α = (1 + ln(2/δ)/2) is treated as a hyperparameter. The arm c chosen is the one whose parameter vector z c maximizes the following: DISPLAYFORM5 The hyperparameter values that are used for all methods across all tasks (the global hyperparameters) are presented in TAB6. The hyper parameters for the low dimensional bandit tasks (we uses the same values for each low dimensional dataset), the paraphrase bandit (multiclass and pool), and the image classification bandit (multiclass and pool) are presented in Table 5. The meanings of the non obvious hyperparameter values are described below:Retrain Epochs: For Sample New and Marginalize; how many times to pass over the new data D t.Update Frequency: How many online rounds to do before updating the rest of the network offline. Num Updates: For Sample All; the number of batches to uniformly sample (and update with) from the entire distribution. Table 5: Method specific experiment details for all tasks. In this section we detail the architectures used for each task. We use the same underlying model for each method. Low dimensional task models. For the Low dimensional tasks, we utilize the same exact Multi Layer Perceptron (MLP) models as used in BID17. The model is a 2 layer MLP with a hidden layer dimension of 100, with ReLu activation functions. Paraphrase task. The paraphrase task uses the same BiMPM architecture and parameters used in. We refer readers to the corresponding paper for details. Image classification task. The convolutions architecture we use is defined as: (i) Two 3 × 3 convolutional layers with 64 filters and ReLu activations; (ii) A 2 × 2 max pooling layer with a stride of 2; (iii) Dropout layer with drop probability 0.25; (iv) A 3 × 3 and 2 × 2 convolutional layer both with 128 filters and ReLu activations; (v) A 2 × 2 max pooling layer with a stride of 2; (vi) Dropout layer with drop probability 0.25; (vii) A fully connected layer with 1024 units, followed by a tanh activation, followed by another fully connected layer with 100 units and a tanh activation. We utilize tanh activations at the end as per , who note that ReLu activations lead to difficulties in estimating the uncertainty.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.