Question: Represent the article for finding a single-sentence TLDR

Question: The detection of out of distribution samples for image classification has been widely researched. Safety critical applications, such as autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution. This paper adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detecting out of distribution pixels, which can localise the unusual objects. It further experimentally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and DeeplabV3+ architectures, as well as proposing a new metric for the task. The evaluation shows that the performance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts. Figure 1: Image from the LostAndFound dataset , where two unlikely objects (storage crates) are almost entirely incorrectly predicted to be road. The Max Softmax method clearly highlights these crates as OOD. (best viewed in colour) Many applications using machine learning (ML) may benefit from out of distribution (OOD) detection to improve safety. When inputs are determined to be out of distribution, the output of an ML algorithm should not be trusted. A large body of research exists for detecting entire images as OOD for the task of image classification. Image-level OOD detection outputs a classification for the entire image; this coarse level of detection may be inadequate for many safety critical applications, including autonomous driving. Most of the pixels in an image taken from an onboard camera will be in distribution (ID), i.e. an image of a road scene with cars, people, and roadway-but an unusual object that was not part of the training set may cause only a small number of OOD pixels. Extending the framework to semantic segmentation networks will allow each pixel to have an "in" or "out of" distribution classification. Applied to autonomous driving, groups of pixels classified as OOD would be considered as unknown objects. Depending on the location of the unknown objects, a planner would then proceed with caution or hand over control to a safety driver. Another application is automatic tagging of images with OOD objects, which would then be sent for human labelling. Figure 1 shows a failure case where OOD detection is beneficial. The two crates are predicted as road. The right image of this figure shows the of pixel-level OOD detection using one of the proposed methods, which clearly identifies the unusual objects. This paper adapts existing state-of-the-art image-level OOD detection methods to the new task of pixel-level OOD classification and compares their performance on a new dataset designed for this task. In addition to adapting the methods, we address the question of whether the best-performing image-level methods maintain their performance when adapted to the new task. In order to answer this question, we also propose pixel-level OOD detection performance metrics, drawing both on existing image-level OOD detection and semantic segmentation performance metrics. Further, we design two new datasets for pixel-level OOD detection with test images that contain both pixels that are in distribution and pixels that are out of distribution, evaluated with two different network architectures-PSPNet and DeeplabV3+ . Somewhat surprisingly, our evaluation shows that the best performing pixel-level OOD detection methods were derived from image-level OOD detection methods that were not necessarily the best performing on the image-level OOD detection task. In summary, the contributions of this paper are the following: • adaptation of image-level OOD detection methods to pixel-level OOD detection and their evaluation; • training and evaluation datasets for pixel-level OOD detection evaluation derived from existing segmentation datasets; and • a new metric for pixel-level OOD detection, called MaxIoU. We use two criteria to select existing image-level OOD detection methods to adapt to pixellevel OOD detection. First, the candidate methods are top performers on image classification datasets. Second, they must be computationally feasible for semantic segmentation. Max Softmax , ODIN , Mahalanobis and Confidence fit both criteria. The Entropy , Sum of Variances (VarSum) , and Mutual Information methods do not meet the first criterion, but are included as an existing uncertainty baseline for the pixel-level OOD classification. An example method that does not meet the second criterion is an ensemble method by , with an ensemble of K leave-out classifiers. In general, images and architectures for semantic segmentation are larger than for image classification, and therefore an ensemble method is much less feasible for segmentation than classification due to GPU memory limitations. Generative Adversarial Networks (GANs) and Auto-Encoders (AEs) are often used in the literature as a method for OOD detection. GANs and AEs are excluded to limit the scope of this work. Table 1 briefly describes the selected image-level OOD detection methods and any architecture modifications necessary to adapt them to OOD detection. 1 The Dim reduction modification is an additional penultimate layer that is a 1 × 1 convolution reducing the depth to 32. DeeplabV3+ has a much larger feature extractor than PSPNet, therefore due to hardware limitation, the Mahalanobis method is not evaluated on this architecture. Each original/adapted method produces a value that can be thresholded to predict whether an image/pixel is OOD. All metrics used in our evaluation are threshold independent, therefore no thresholds are discussed in this section. See Appendix A for more detailed description of each OOD detection method. to OOD data prior to evaluation. This unfair comparison has been criticised in the past . Most methods have a input perturbation step; however, this is removed as it also requires access to OOD data before evaluation. Previous work on image-level OOD detection designates a dataset used for training as the ID dataset, and the dataset for testing as the OOD dataset. This framework is not easily extended to semantic segmentation, as any two datasets may share features with the ID dataset. For example, people may exist in both datasets, but one is indoor scenes and the other is outdoor scenes. attempt to get around this issue by inserting animals from the COCO dataset or internet images into the Cityscapes dataset. The lack of realism of the inserted images (improper scale and hard boundaries) make the created dataset insufficient for OOD detection. The remainder of this section describes the datasets used in this work and any modifications required. The dataset used for training the weights of all networks is the unmodified Cityscapes dataset . It is the most prominent dataset for training and benchmarking semantic segmentation for road scene images. The diverse SUN dataset and the India Driving Dataset (IDD) are used as the main evaluation datasets. The main motivation for using the SUN dataset is that it has a large variety of scenes (e.g. street, forest, and conference room), as well as a large variety of labels (e.g. road, door, and vase). In total there are 908 scene categories and 3819 label categories. Anonymous label submissions are ignored, as their validity is not confirmed. However, the SUN dataset is not very realistic in terms of what a vehicle camera would see. The IDD dataset is an autonomous vehicle dataset that has all the same labels of Cityscapes with the addition of the autorickshaw class. Although the classes are the same, the instances tend to be OOD, due to different building architecture, road infrastructure, etc. This shift in features leads to higher average values OOD predictions as shown in Figure 2. Therefore only the car class as ID and auto-rickshaw class as OOD are used for evaluation. The SUN dataset labels have to be modified before the dataset can be used for evaluating OOD detection. 2 The approach for modifying this dataset is similar to the open world dataset created by. Let C = {person, car, ignore, ...} be the set of labels available in Cityscapes. Let S = {person, roof, chair, ...} be the set of labels available in the SUN dataset. Ambiguous classes such as "wall" (can appear indoor and outdoor) or "path" (a sidewalk or a dirt path in the woods) are sorted into a third set A ⊂ S. The map M: S → C ∪ {OOD} is defined as: For every image in the SUN dataset, each pixel label l i gets a new label l i = M(l i). Pixels with the "ignore" class are given a weight of 0 during evaluation. All the images in the SUN dataset have various dimensions. To prevent artefacts in the input after resizing, only images that have more than 640 · 640 = 409, 600 pixels are used. All images are resized to the same size as Cityscapes (1024 × 2048). The SUN dataset is split into a train set and an evaluation set (25%/75% split). It is stressed that the training set is used only to select the best hyperparameters of the ODIN method. Following previous work on image-level OOD detection, a synthetic dataset of random normally distributed images are used as well. The random normal noise is usually very easily detected by all methods, therefore Perlin noise images are used as well. Perlin noise is a smooth form of noise that has large "blobs" of colour that is much harder to detect and filter. Each of these datasets have the entire image labelled as OOD. All OOD datasets used are mixed with Cityscapes evaluation sets. The ing ratio of ID to OOD pixels for IDD/Cityscapes and SUN/Cityscapes is about 7.5:1 and 3:1 respectively. Since ODIN requires ODD data for hyperperameter tuning, Cityscapes training set is mixed with a held-out training set of OOD data in the same manner as the evaluation sets. There are five metrics used to evaluate the performance of each model. The first four listed below are the same metrics usually used by previous works on OOD detection (; ; ;). Since the output of semantic segmentation is so much larger than image classification, the below metrics must be approximated. This is done by using 400 linearly spaced thresholds between 0 and 1 and tracking all true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) for each threshold. Each pixel contributes to one of TP, TN, FP, FN for a given threshold, based on the ground truth and OOD prediction value -accumulated across all images. • AUROC -Area under the receiver operating characteristic (ROC) curve. The ROC curve shows the false positive rate (FPR) FP FP+TN against the true positive rate (TPR) TP TP+FN. The area under this curve is the AUROC metric. • AUPRC -Area under the precision recall (PR) curve. The PR curve shows the precision TP TP+FP against the TPR (or recall). The area under this curve is the AUPRC. • FPRatTPR -FPR at 95% TPR. Extracted from the ROC curve, this metric is the FPR value when the TPR is 95%. • MaxIoU -Max intersection over union (IoU). IoU is calculated by TP TP+FP+FN. MaxIoU is the maximum IoU over all thresholds. A common metric in the literature, coined detection error , is excluded as it is a linear transformation of the FPRatTPR metric. Therefore, it adds no useful information. The inspiration for MaxIoU issued from the semantic segmentation community. Mean intersection over union (mIoU) is the canonical performance metric for semantic segmentation. MaxIoU is similar, but targets tasks that are threshold dependant. Thresholds selected by MaxIoU punish false positives more than AUROC. The optimal threshold is generally greater, ing in fewer positive predictions than for AUROC. To verify that the MaxIoU is complimentary to AUROC, the optimal thresholds selected by each were experimentally compared. The mean absolute difference between the threshold selected via Youden index and that chosen for MaxIoU was found to be 0.039. PSPNet and DeeplabV3+ network architectures are used with the Resnet and Xception feature extractors respectively. The two driving factors for these networks are: near top performance on the Cityscapes benchmark (2.6 and 1.7 less mIoU than the top scoring algorithm) and the final operation before the softmax classification layer being a bi-linear upsample. The upsample ensures that any method's OOD prediction will be directly correlated with the pixel prediction from the softmax layer, as most methods rely on the softmax layer. There is a clear relationship between any spatial location in the penultimate layer and an output pixel. The Xception feature extractor is much larger than the Resnet -therefore due to hardware limitations, the space intensive Mahalanobis method is only evaluated on PSPNet. There are two main research questions that this paper focuses on. Each question has an associated experiment. • RQ1: Do the required modifications to architecture and loss functions negatively affect the semantic segmentation performance of the network? • RQ2: Which OOD detection method performs the best? To answer RQ1, we evaluate the semantic segmentation performance on Cityscapes using the standard class mean intersection over union (mIoU) metric. As long as the performance drop of modified networks is not too large, the modifications do not interfere with the original task. RQ2 is answered by evaluating each method on the datasets described in Section 3. Figure 3 shows the comparison of the performance of the different methods using the PSPNet architecture. Each graph shows a different metric (c.f . Section 4) for each dataset. Max Softmax, ODIN, and Mahalanobis follow the same trend as their image classification counterparts with the SUN dataset, increasing in performance in that order. However, for the IDD dataset the order is Mahalanobis, Max Softmax, then ODIN, in increasing performance. For image-level OOD detection, the Confidence method outperforms the Max Softmax baseline. However, for pixel-level OOD detection it is worse. For real datasets, VarSum is worse than Mutual Information, but better on the synthetic datasets. Across each metric, VarSum has the biggest performance increase from the modified SUN and IDD datasets to the random normal dataset, moving from worst to near top performance. This however is not the same for the Perlin noise dataset, as the low frequency noise is not easily filtered. The Confidence method seems to mostly learn to highlight class boundaries, as that is where prediction errors are likely to occur. Therefore the prediction loss force lower confidence levels. This makes it less suitable for the random normal and Perlin noise datasets, where the network predicts one single class for the majority of the input. Figure 4 shows the comparison of the performance of the different methods using the DeeplabV3+ architecture. With respect to AUROC, AUPRC, and MaxIoU on the IDD dataset Mutual Information Figure 5: Comparison of the Mutual Information method on an IDD dataset image, successfully predicted. The top row is using the PSPNet architecture, the bottom row is using the DeeplabV3+ architecture. The columns from left to right are: input image, ground truth, class prediction (Cityscapes colours), ODD prediction. The OOD prediction is masked to only cars and auto-rickshaws. Best viewed in colour. has the best performance, followed by ODIN. On all datasets and metrics, VarSum has the worst or is near worst performance. Confidence has much better performance with the DeeplabV3+ architecture than the PSPNet architecture, especially on the random datasets. The for DeeplabV3+ are much less conclusive than PSPNet as the relative ordering across the different metrics changes significantly. The metrics values reported in the original works for the image-level OOD detection for all methods with image-level OOD detection counterparts are very close to 1 (∼0.95 and above). However, in our experiments, the are much less than 1. One peculiar is the significant difference in performance between the Confidence method on DeeplabV3+ and PSPNet on the random normal and Perlin noise datasets. ODIN performs well on all datasets and architectures. The performance of this method is aided by its access to OOD data during hyperparameter tuning, however. Figure 5 shows a comparison of the Mutual Information method using both the PSPNet and DeeplabV3+ architecture. Mutual information is one of the worst performing on PSPNet and the best performing on DeeplabV3+. There are two major observations to note. The first is that both networks label different parts of the auto-rickshaw as different classes. The second is the increase in OOD prediction of all cars from DeeplabV3+ to PSPNet, while the OOD prediction of the autorickshaw remains approximately the same. Both architectures successfully predict the majority of the pixels correctly; however, the separation is better for DeeplabV3+. Figure 6 shows a comparison of the Entropy method using both the PSPNet and DeeplabV3+ architecture. The auto-rickshaw is mistaken by both architectures as a car, therefore the ing OOD prediction is similar to other cars in the scene. This failure highlights a more general failure of pixel-level OOD detection methods. When the network is confident about a wrong prediction, the OOD methods are fooled as well. The drop in performance for pixel-level OOD detection is likely due to features that cause large disruptions at the pixel-level, but would not affect an entire image; for example, shadows, occlusion, and far away objects. Figure 7 shows an example of shadows and far away objects in the bottom row. At the end of the road, most pixels are high OOD values as well as the right side of the scene, which is in the shade of a building. The top row of Figure 7 shows an interesting failure case of a flooded road being predicted as road with a low OOD value. As can be seen in all example outputs, class boundaries are highlighted. A classical computer vision algorithm was developed, using a series of erosion, dilation and other filters to remove these boundaries. In general performance was increased; however, the increase was on the order of 10 −3. To our knowledge, there are very few works researching pixel-level OOD detection. create a dataset by overlaying animals and objects from the COCO dataset on top of the Cityscapes dataset , this however lacks realism. This new dataset is tested with various OOD detection methods. train a segmentation network with two datasets-one ID and one OOD dataset. The network learns to classify between the two on a per pixel-level. This is compared to the Max Softmax baseline. The major flaw in this method is that the network learns its weights from OOD samples. There are some commonalities to Active Learning for semantic segmentation . These studies attempt to use some form of output uncertainty to choose which images are best for training/labelling in order to reduce the number of training examples needed. They produce a heat map similar to the OOD detection output. These heat maps are then aggregated across a whole image to produce a score for the entire image. create an open world dataset. Known object labels are drawn from the COCO dataset , and labels drawn from the NYU dataset are relabelled as unknown if the class doesn't exist in COCO. This methodology is very similar to the modified SUN dataset in Section 3. A generic object instance level segmentation algorithm is developed, based on a class specific object detector, a boundary detector and simulated annealing, and is evaluated on the new dataset. This approach splits the image into visually distinct connected regions, but is too slow for real-time applications. Several methods for detecting OOD pixels were adapted from image-level OOD detection, as well as a pixel uncertainty estimation. These methods were compared using metrics previously established by OOD detection works, as well as a new metric that has roots in the semantic segmentation task. This paper also contributed two new datasets for pixel-level OOD classification derived from semantic segmentation datasets that have common classes but also unique ones. There is great room for improvement for pixel-level OOD detection. One shortcoming for all the methods compared in this paper is the ability to distinguish between class boundary pixels and OOD pixels. We tested classical computer vision techniques that could be used to visually fix this problem, but the performance increase was negligible. The ODIN and Mahalanobis methods have the best performance with PSPNet and SUN dataset, beating the VarSum, Mutual Information, and Confidence methods by a significant margin. However, Mutual Information has the best performance with DeeplabV3+ and the IDD dataset, with the other methods following closely. Therefore the ODIN, Mahalanobis, and Mutual Information methods should be considered the baseline for further research in pixel-level OOD detection. Understanding the faults of pixel-level OOD detectors is crucial for progress. This would include categorising the failure cases of a detector. For example, understanding why a flooded road is not highlighted, and what makes that different to shadows falsely being highlighted. Each subsection uses notation close to the original works, so that readers can refer to said works for more detail. The appendix also provides additional detail. Notation should not be carried between sections unless explicitly stated. Assume the neural network is a function f, and input image is x. The set of pixels P will be used with typical subscripts i, j ∈ P. For example f (x) i is the i th pixel of the of f (x). Note that most methods have a input perturbation step. They are included here for completeness, however, they are not used in any evaluation as it requires access to OOD data. This access to OOD data has been criticised as an unfair comparison to methods that do not require access to OOD data. At test time, dropout can be used as an estimate for model uncertainty. A multiplier of 0 or 1 is randomly chosen for each neuron at test time. use the variation in predictions of a model that uses dropout at test time to compute an estimate of model uncertainty. The output is a variance per class, the sum of these variances is the estimate of model uncertainty. This is performed per pixel to get the value s i for pixel i ∈ P. A.2 show that the max softmax value can be used to detect OOD examples as a baseline for image classification. The softmax function is defined as: Where Sŷ(x) is the max softmax value andŷ is used to signify the chosen index. Given a prediction p i = f (x) i, the max softmax value for each pixel i is v i = Sŷ i (p i), and that value is used to determine if that pixel is OOD. A.3 create a similar method to the max softmax value, dubbed ODIN. This method adds temperature scaling and input preprocessing. The softmax function in Equation 3 is modified to include a temperature value T: Liang et al. found that perturbing the input in the direction of the gradient influences ID samples more than OOD samples, thus separating ID and OOD examples more. The input preprocessing step is: where is a hyperparameter chosen from a set of 21 evenly spaced values starting at 0 and ending at 0.004. The best temperature value is chosen from a predefined set of temperatures {1, 2, 5, 10, 20, 50, 100, 200, 500, 1000}. The temperature-scaled and preprocessed max softmax score v i = Sŷ(x; T) i for pixel i is used to predict if that pixel is OOD. A.4 use the Mahalanobis distance for detecting OOD samples. The Mahalanobis distance is the number of standard deviations a vector is away from the mean, generalised to many dimensions. Each feature vector at a spatial location in the penultimate layer is assumed to be normally distributed. Each distribution is parameterised by the pixel class mean µ ci and global class covariance Σ c for pixel i and class c. The global class covariance is computed for all pixels of a given class, independent of their location. Initial tests showed that using pixel class means and a global class covariance has better performance than global or pixel class mean and pixel class covariance, therefore they are used throughout. The labels are resized to match the height and width of the penultimate layer using nearest neighbour interpolation. The two quantities µ ci and Σ c are computed as follows: where N c is the number of examples of class c, and X j is the j th example of the training dataset X. f l is the output of the l th layer of the network f. Here l is the second to last layer. Each spatial location has a class distance and minimum distance computed by: This increases the number of matrix multiplications required to compute each pixel distance. Due to hardware memory limitations, a dimensionality reduction layer is needed after the penultimate layer, reducing the depth from 512 to 32 via a 1 × 1 convolution. An input preprocessing step is also performed. The new inputx is computed by: Instead of a logistic regression layer, the minimum distance is normalised to have zero mean and unit variance. The sigmoid function σ is applied to clamp to the interval: where µ and s are the mean and standard deviation of all M (x) computed over the whole training dataset. v i is used to determine if pixel i is OOD. The prediction values are resized, with bi-linear interpolation, to the original input size. train a secondary branch of an image classification network to output a single confidence value. This confidence value is learned via a regularization loss. The loss gives "hints" to the network when the confidence is low, and it penalizes low confidence. Similar to the image classification method, a secondary branch is added to the network f. Therefore c i, p i = f (x) i, where c i is the confidence value and p i is the prediction vector for pixel i ∈ P. The new branch is trained by creating a new prediction p i as: where B is a Bernoulli distribution and y i is the one hot ground truth vector for pixel i. The mean negative log likelihood is then applied to the new p i as well as a regularization term is added to force each c i to 1 (i.e. high confidence): where L is the total loss that is used to train the network, and λ is a hyperparameter. λ is 0.5 in all experiments. At test time a preprocessing step is applied to each pixel, and is computed using the gradients of the L c loss. Note that the adapted L c sums over all the pixel confidence predictions c i, therefore the gradient implicitly sums over output pixels as well. Letp i,c i = f i (x), then v i =c i is used to determine if a pixel is OOD. A.6 ENTROPY Shannon entropy is an information theoretic concept, that is used to determine how much information a source contains. The entropy equation is H: R n → R: Since H was developed for probabilities, x must behave like a probability distribution meaning: ∀i, x i >= 0 train an image-level classifier with a third outlier dataset that is disjoint from both the training and OOD dataset. An auxiliary loss is added minimising the entropy of predictions of outlier images. The network learns to predict uniform values for all classes. The max softmax value is then used at test time to determine if a sample is OOD. The entropy function is applied to the softmax output, which satisfies the properties in Equations 21 and 22. Let P = f (x) then v i is used to determine if pixel i is OOD. develop Bayesian uncertainty estimation methods and evaluation metrics to evaluate if higher uncertainty is correlated with higher error rates. The estimation can be used for OOD detection as well, similar to the Entropy and VarSum methods. There are two quantities required. Predictive entropy: and Aleatoric Entropy: In both equations, x is the input and y is the prediction. ω i is the set of weights selected by dropout at iteration i. Mutual information is then MI(y|x) is used to determine if a pixel is OOD or ID.

Represent the TLDR for finding the corresponding scientific article

Evaluating pixel-level out-of-distribution detection methods on two new real world datasets using PSPNet and DeeplabV3+.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the following document

Question: We present a domain adaptation method for transferring neural representations from label-rich source domains to unlabeled target domains. Recent adversarial methods proposed for this task learn to align features across domains by ``fooling'' a special domain classifier network. However, a drawback of this approach is that the domain classifier simply labels the generated features as in-domain or not, without considering the boundaries between classes. This means that ambiguous target features can be generated near class boundaries, reducing target classification accuracy. We propose a novel approach, Adversarial Dropout Regularization (ADR), which encourages the generator to output more discriminative features for the target domain. Our key idea is to replace the traditional domain critic with a critic that detects non-discriminative features by using dropout on the classifier network. The generator then learns to avoid these areas of the feature space and thus creates better features. We apply our ADR approach to the problem of unsupervised domain adaptation for image classification and semantic segmentation tasks, and demonstrate significant improvements over the state of the art. Transferring knowledge learned by deep neural networks from label-rich domains to new target domains is a challenging problem, especially when the source and target input distributions have different characteristics. Such domain shifts occurs in many practical applications. For example, while simulated driving images rendered by games provide a rich source of labeled data for semantic segmentation BID19, deep models trained on such source data do not transfer well to real target domains (FIG0). When target-domain labels are unavailable for fine-tuning, unsupervised domain adaptation must be applied to improve the source model. Recent methods for unsupervised domain adaptation attempt to reduce the discrepancy between the source and target features via adversarial learning BID28; BID4 ). They divide the base network into a feature encoder G and classifier C, and add a separate domain classifier (critic) network D. The critic takes the features generated by G and labels them as either source-or target-domain. The encoder G is then trained with an additional adversarial loss that maximizes D's mistakes and thus aligns features across domains. However, a major drawback of this approach is that the critic simply predicts the domain label of the generated point and does not consider category information. Thus the generator may create features that look like they came from the right domain, but are not discriminative. In particular, it can generate points close to class boundaries, as shown in FIG0 (e), which are likely to be misclassified by the source model. We argue that to achieve good performance on the target data, the adaptation model must take the decision boundaries between classes into account while aligning features across domains (FIG0). Moreover, since our setting is unsupervised adaptation, this must be accomplished without labels on target data. In this paper, we propose a novel adversarial alignment technique that overcomes the above limitation and preserves class boundaries. We make the following observation: if the critic could detect points near the decision boundary, then the generator would have to avoid these areas of the feature space in order to fool the critic. Thus the critic would force the generator to create more discriminative features. How can we obtain such a critic? If we alter the boundary of the classifier C slightly and measure the change in the posterior class probability p(y|x), where y and x denote class and We propose to use the boundary information to achieve low-density separation of aligned points.input respectively, then samples near the decision boundary are likely to have the largest change. In fact, this posterior discrepancy is inversely proportional to the distance from the class boundary. We thus propose to maximize this posterior discrepancy to turn C into a critic sensitive to nondiscriminative points. We call this technique Adversarial Dropout Regularization. Here, dropout is not used in the standard way, which is to regularize the main classifier and make it insensitive to noise. Instead, we use dropout in an adversarial way, to transform the classifier into a critic sensitive to noise. Compared to previous adversarial feature alignment methods, where the distributions p(x) are aligned globally, our method aligns target features away from decision boundaries, as illustrated in FIG0 (f).Our ADR approach has several benefits. First, we train the generator G with feedback from the classifier C, in contrast to existing methods, which use an unrelated critic D. Second, our method is general and straightforward to apply to a variety of domain adaptation problems, such as classification and semantic segmentation. Finally, since ADR is trained to align distributions, it is also applicable to semi-supervised learning and training of generative models, such as Generative Adversarial Networks (GANs) BID6 ). Through extensive experiments, we demonstrate the benefit of ADR over existing domain adaptation approaches, achieving state-of-the-art in difficult domain shifts. We also show an application to semi-supervised learning using GANs in appendix. Domain Adaptation. Recent unsupervised domain adaptation (UDA) methods for visual data aim to align the feature distributions of the source and target domains; BID28; BID5 Long et al. (2015b); BID31 ). Such methods are motivated by theoretical stating that minimizing the divergence between domains will lower the upper bound of the error on target domain BID0 ). Many works in deep learning utilize the technique of distribution matching in hidden layers of a network such as a CNN BID28; BID5 Long et al. (2015b) ). However, they measure the domain divergence based on the hidden features of the network without considering the relationship between its decision boundary and the target features, as we do in this paper. Low-density Separation. Many semi-supervised learning (SSL) methods utilize the relationship between the decision boundary and unlabeled samples, a technique called low-density separation (; BID12). By placing the boundary in the area where the unlabeled samples are sparse, these models aim to obtain discriminative representations. Our method aims to achieve low-density separation for deep domain adaptation and is related to entropy minimization for semi-supervised learning BID8 ). used entropy minimization in their approach to directly measure how far samples are from a decision boundary by calculating entropy of the classifier's output. On the other hand, our method tries to achieve low-density separation by slightly moving the boundary and detecting target samples sensitive to the movement. As long as target samples features are robust to the movement, they will be allowed to exist relatively nearby the boundary compared to source samples, as FIG0 shows. In entropy minimization is only a part of the overall approach. To compare our ADR approach to entropy minimization more directly, we use a new baseline method. To our knowledge, though this method has not been proposed by any previous work, it is easily achieved by modifying a method proposed by BID23 ). For this baseline, we train a model that generates features to minimize the entropy of the output probability for target samples. The details of the baseline are provided in appendix. In short, the generator tries to minimize the entropy of the target samples, whereas the critic tries to maximize it. The entropy is directly measured by the output of the classifier. This baseline is similar to our approach in that the goal of the method is to achieve low-density separation. Dropout. Dropout is a method that prevents deep networks from overfitting BID24 ) by randomly dropping units from the neural network during training. Effectively, dropout samples from an exponential number of different thinned networks at training time, which prevents units from co-adapting too much. At test time, predictions are obtained by using the outputs of all neurons. If the thinned networks are able to classify the samples accurately, the full network will as well. In other words, dropout encourages the network to be robust to noise. In our work, we use dropout to regularize the feature generation network G, but in an adversarial way. We train the critic C to be sensitive to the noise caused by dropout and use C to regularize G so that it generates noise-robust features. To our knowledge, this use of dropout is completely different from existing methods. We assume that we have access to a labeled source image x s and a corresponding label y s drawn from a set of labeled source images {X s, Y s}, as well as an unlabeled target image x t drawn from unlabeled target images X t. We train a feature generation network G, which takes inputs x s or x t, and a network C that acts as both the main classifier and the critic. When acting as the classifier, C takes features from G and classifies them into K classes, predicting a K-dimensional vector of logits {l 1, l 2, l 3 ...l K}. The logits are then converted to class probabilities by applying the softmax function. Namely, the probability that x is classified into class j is denoted by p(y = j|x) = DISPLAYFORM0 We use the notation p(y|x) to denote the K-dimensional probabilistic output for input x. When C is acting as the critic, we want it to detect the feature encodings of target samples near the decision boundary. We propose to make C sensitive to such samples by slightly perturbing its decision boundary and measuring the change in the posterior class probability p(y|x). This change is likely to be largest for samples near the decision boundary. The network C is then trained to increase this change, while the feature generation network G is trained to decrease it. Through this adversarial training, G learns to'fool' the critic and generate target features far away from the decision boundary, thus avoiding ambiguous features. The weights of G can be initialized either by pre-training on some auxiliary dataset (e.g., ImageNet), or with random weights, while C uses random initialization. In the next section, we show how we utilize dropout to perturb the boundary in the critic and measure sensitivity. We then show the training procedure of our method. Finally, we give some intuition behind adversarial dropout and improve our method based on this insight. Consider the standard training of a neural network using dropout. For every sample within a minibatch, each node of the network is removed with some probability, effectively selecting a different classifier for every sample during training. We harness this idea in a very simple way. We forward input features G(x t) to C twice, dropping different nodes each time and obtaining two different output vectors denoted as C 1 (G(x t)), C 2 (G(x t)). In other words, we are selecting two different classifiers C 1 and C 2 from C by dropout as in Fig. 2. In the figure, the corresponding posterior probabilities are indicated as p 1 (y|x t), p 2 (y|x t), abbreviated as p 1 and p 2 in the following discussion. In order to detect the change of predictions near the boundary, the critic tries to increase the difference between the predictions of C 1 and C 2. This difference corresponds to C's sensitivity to the noise caused by dropout. To measure the sensitivity d(p 1, p 2) between the two obtained probabilistic outputs, we use the symmetric Kullback Leibler (KL) divergence. Formally, the divergence is calculated as DISPLAYFORM0 Update G to minimize sensitivity on target inputs (Fix C) Update C to maximize sensitivity on target inputs (Fix G) Train G, C on source inputs using classification loss DISPLAYFORM0 Figure 2: Overview of ADR. Left: We train G, C with classification loss on source and sample a critic consisting of two classifiers using dropout. The critic's sensitivity is measured as the divergence between the class predictions of C 1 and C 2 on the same input. Right: Adversarial training iterates two steps: the critic tries to maximize the sensitivity while the generator tries to minimize it.where KL divergence between p and q is denoted as D kl (p|q). In our approach, C works as both critic and classifier. The following three requirements are imposed by our method: 1) C and G must classify source samples correctly to obtain discriminative features; 2) C should maximize the sensitivity for target samples to detect the samples near the boundary; 3) G should learn to minimize the sensitivity to move target samples away from the boundary. The training within the same mini-batch consists of the following three steps. Step 1, in this step, C is trained as a classifier. C and G have to classify source samples correctly to obtain discriminative features. Thus, we update both networks' parameters based on the following standard classification loss. Given source labels y s and samples x s, the objective in this step is DISPLAYFORM0 C(G(x s)) k returns the probability that the sample x s is assigned to class k. Step 2, in this step, C is trained as a critic to detect target samples near the boundary. Two classifiers are sampled from C for each target sample using dropout twice to obtain p 1 and p 2. Then, C's parameters are updated to maximize the sensitivity as measured by Eq. 1. Since C should learn discriminative features for source samples, in addition to the sensitivity term, we add Eq. 2. We experimentally confirmed that this term is essential to obtain good performance. DISPLAYFORM1 DISPLAYFORM2 C 1 and C 2 are sampled from C randomly. Step 3, in order to obtain representations where target samples are placed far from the decision boundary, G is trained to minimize sensitivity. Here we do not add the categorical loss for source samples as in Step 2, as the generator is able to obtain discriminative features without it. DISPLAYFORM3 We update the parameters of C and G in every step following the defined objectives. We experimentally found it beneficial to repeat Step 3 n times for each mini-batch. show the decision boundary obtained by keeping one neuron in the last hidden layer and removing the rest. Red points are source samples of class one, green points are class two. Black points are target samples. The yellow region indicates where the samples are classified as class one, cyan region class two. We see that the neurons do not learn very diverse features. Column 6 shows the boundary obtained by keeping all 5 neurons. Bottom row: Boundaries learned by the model adapted by our adversarial dropout method. Unlike the top row, here neurons 3,4,5 learn diverse features which in diverse boundaries. Our ADR approach encourages different neurons of the classifier to learn different characteristics of the input (see Sec. 4.1.) The output is the combination of shared and unshared nodes, therefore, to maximize the sensitivity, the unshared nodes must learn different features of target samples. As learning proceeds, each neuron in C will capture different characteristics. At the same time, to minimize the sensitivity, G learns to extract pure categorical information. If G outputs features which are not related to categorical information, such as texture, slight contrast or difference of color, C will utilize them to maximize sensitivity. The trained classifier will be sensitive to the perturbation of targets caused by dropout. We note that our approach is contrary to methods called adversarial example training BID7; BID15 ) which train the classifier to be robust to adversarial examples. They utilize input noise which can deceive or change the output of the classifier, and incorporate it to obtain a good classifier. Our ADR method encourages the feature generator to obtain noise-robust target features. However, with regard to the classifier, it is trained to be sensitive to noise. To improve the final accuracy, we learn another classifier C that is not trained to be sensitive to the noise. C takes features generated by G and is trained with classification loss on source samples. The loss of C is not used to update G. We compare the accuracy of C and C in experiments on image classification. 4.1 EXPERIMENT ON TOY DATA Experimental Setting. In this experiment, we observe the decision boundary obtained by each neuron to demonstrate that ADR encourages the neurons to learn different input characteristics. We use synthetic "two moons" data for this problem. Two dimensional samples from two classes are generated as source samples. Target samples are obtained by rotating the source samples. In our setting, the rotation was set to 30 degrees and data was generated with scikit-learn BID17 ). We train a six-layered fully-connected network; the lower 3 layers are used as feature generator, and upper 3 layers are used as classifier. We used Batch Normalization BID11 ) and ReLU as activation function. The number of neurons are for feature generator, for classifier. We visualize the boundary obtained from each neuron in the last layer by removing the output of all other neurons. Results. We show the learned boundary in FIG1. In the baseline model trained only with source samples (top row), two of five neurons do not seem to learn an effective boundary, and three neurons learn a similar boundary. On the other hand, in our method (bottom row), although two neurons do not seem to learn any meaningful boundary, three neurons learn distinctive boundaries, demonstrating greater diversity. Each neuron is trained to be sensitive to the noise caused by target samples. The final decision boundary (rightmost column) classifies most target samples correctly. The accuracy of our proposed method is 96% whereas the accuracy of the non-adapted model was 84%. Experiments on Digits Classification. We evaluate our model on adaptation between digits datasets. We use MNIST BID14 ), SVHN BID16 ) and USPS datasets and follow the protocol of unsupervised domain adaptation used by BID29 ). To extensively compare our method with previous methods, in adaptation from MNIST to USPS, we applied our method to a different protocol used in. We assume no labeled target samples and use fixed hyper-parameters for all experiments, unlike other works that use a target validation set BID20 ). The number of iterations forStep 3 was fixed at n = 4. We used the same network architecture as in BID29 ), but inserted a Batch Normalization layer before the activation layer to stabilize the training. We used Adam BID13 ) for optimizer and set the learning rate to 2.0 × 10 −4, a value commonly reported in the GAN literature. We compare our approach to several existing methods and to the entropy minimization baseline (ENT) obtained by modifying (Springenberg FORMULA6). As we mentioned in Section 2, this is a model that generates features to minimize the entropy of the output probability for target samples. Due to space limitations, we provide a detailed explanation of this baseline in the appendix. Results in Table 1 demonstrate that ADR obtains better performance than existing methods. In particular, on the challenging adaptation task from SVHN to MNIST, our method achieves much better accuracy than previously reported. FIG2 shows the learning curve of each experiment. As sensitivity loss increases, the target accuracy improves. This means that as critic C learns to detect the non-discriminative samples, feature generator G learns to fool it, ing in improved accuracy. In addition, we can see that the sensitivity of source samples increases too. As mentioned in Sec 3.3, the critic network should learn to capture features which are not very important for classification, such as texture or slight edges, and it seems to also capture such information in source samples. The accuracy of the classifier C (denoted by red), which is trained not to be sensitive to the noise, is almost always better than the accuracy of the critic network. In adaptation from SVHN to MNIST FIG3 ), the accuracy of the critic often suffers as it becomes too sensitive to the noise caused by dropout. On the other hand, the accuracy shown by the red line is stable. Our ENT baseline shows good performance compared to other existing methods. This indicates the effectiveness of methods based on entropy minimization. In FIG3, we compare our proposed method and ENT in terms of entropy of target samples. Our method clearly decreases the entropy, because target samples are moved away from the decision boundary. Yet, its behavior is different from ENT. Interestingly, the entropy is made smaller than ENT in case of adaptation from USPS to MNIST FIG2 ) though ENT directly minimizes the entropy and our method does not. On the SVHN to MNIST task FIG2 ), the entropy of ADR is larger than ENT, which indicates that our method places the target samples closer to the decision boundary than ENT does. Experiments on Object Classification. We next evaluate our method on fine-tuning a pretrained CNN. We use a new domain adaptation benchmark called the VisDA Challenge BID18 ) which focuses on the challenging task of adapting from synthetic to real images. The source domain consists of 152,409 synthetic 2D images from 12 object classes rendered from 3D models. The validation and test target domains consists of real images, which belong to the same classes. We used the validation domain (55,400 images) as our target domain in an unsupervised domain adaptation setting. We evaluate our model on fine-tuning networks pretrained on ImageNet BID2 ): ResNet101 ) and ResNext BID30 ). For the feature generator, we use the pretrained CNN after removing the top fully connected layer. For the classification network, we use a three-layered fully connected network. Table 2 shows that our method outperformed other distribution matching methods and our new baseline (ENT) in finetuning both networks by a large margin. ENT did not achieve better performance than existing methods, though improvement over the source only model was observed. Although this method performed well on digits, it does not work as well here, possibly because of the larger shift between very different domains. In the experiment on ResNext, after training G and C, we retrained a classifier C just on the features generated by G due to GPU memory limitations, and observed improvement in both networks. Image Segmentation experiments. Next, we apply our method to adaptation for semantic image segmentation. Image segmentation is different from classification in that we classify each pixel in the image. To evaluate the performance on segmentation, the synthetic GTA5 (Richter et al. FORMULA7) dataset is used as source, and real CityScape dataset is used as target. Previous work tackled this problem by matching distributions of each pixel's feature in a middle layer of the network BID10 ). In this work, we apply ADR by calculating sensitivity between all pixels. The training procedure is exactly the same as in classification experiments. We use the ResNet50 pretrained on ImageNet, and utilize an FCN (Long et al. (2015a) ) based network architecture. Further, we utilize the more recent Dilated Residual Networks (DRN) 105 layered model BID32 ), which outperforms ResNet50 on a semantic segmentation task. Table 2: Results on Visda2017 classification datasets BID18 ). DANN and MMD are distribution alignment methods proposed by BID4 ) and (Long et al. (2015b) ) respectively. Ours (retrain classifier) means the classifier retrained for our proposed generator as we mentioned in Sec 3.3. Our proposed method shows much better performance than existing methods. For the feature generator, we use the pretrained network without fully-connected layers. For the classifier, we use a fully-convolutional network with dropout layers. Due to limited memory, the batch size is set to 1. We include details of the network architecture in appendix. For comparison, we train a domain classifier based model for our network (DANN). We build a domain classifier network for the features of each pixel following BID10 ).In Table 3, we show the qualitative comparison with existing methods. ADR clearly improves mean IoU (Intersection-over-Union) compared to the source-only and competing models, beating state-ofthe-art by a large margin. When we apply ADR to DRN, the accuracy improves much more than for ResNet50, and is 12.4 points higher than the model trained only on GTA5 source samples. This is likely because ADR exploits the strong representation of the pretrained DRN network. Although we implemented ENT in this setting, the accuracy was much worse than the Source Only model with a mIoU of 15.0 in training ResNet50. The ENT method does not seem to work well on syntheticto-real shifts. Finally, we illustrate our method's improvement on example input images, ground truth labels, images segmented by the Source Only model and our method in FIG7. While the Source Only model seems to suffer from domain shift, ADR generates a clean segmentation. These experiments demonstrate the effectiveness of ADR on semantic segmentation. In this paper, we introduced a novel approach for aligning deep representation, Adversarial Dropout Regularization, which learns to generate discriminative features for the target domain. The method Table 3: Results on adaptation from GTA5 → Cityscapes. DANN and FCN Wild denote methods proposed by BID4 ) and BID10 respectively. consists of a critic network that can detect samples near the task decision boundary and a feature generator that fools the critic. Our approach is general, applies to a variety of tasks, and does not require target domain labels. In extensive domain adaptation experiments, our method outperformed baseline methods, including entropy minimization, and achieved state-of-the-art on three datasets. We also show how to apply our method to train Generative Adversarial Networks for semisupervised learning in the appendix. Our method aims to move target samples away from the decision boundary. Some techniques used in training Generative Adversarial Networks can be applied to achieve our goal too. BID23; BID21 ) used small number of labeled samples to train critic. Critic is trained to classify real samples into K classes. They also trained critic to move unlabeled real images away from the boundary by minimizing entropy of the critic's output. Generated fake images are moved near the boundary by maximizing the entropy. On the other hand, generator is trained to generate fake images which should be placed away from the boundary. This kind of method can be easily applied to domain adaptation problem. We would like to describe the method along with our problem setting. Similar to our method, we have critic networks C and generator G. C classifies samples into K class. C is trained to maximize the entropy of target samples, which encourages to move the target samples near the boundary. Then, G is trained to minimize the entropy of them. Thus, G tries to move target samples away from the boundary. The only difference from our method is that we used entropy term for adversarial training loss. That is, in this method, we replace our sensitivity term d(p 1, p 2) in Eq. 4 with entropy of the classifier output. The adversarial loss for this baseline method is a following one. DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 The hyper-parameter n, how many times we update G for adversarial loss in one mini-batch, is set as n = 4. Experimentally, it worked well for all settings. We follow the protocol used in BID29 ). For adaptation from SVHN to MNIST, we used standard training splits of each datasets as training data. For evaluation, we used test splits of MNIST. For the adaptation between MNIST and USPS (P1), we sampled 2000 images from MNIST and 1800 images from USPS. For the adaptation between MNIST and USPS (P2), we used all training images of MNIST and USPS following. In these experiments, we composed the mini-batch half from source and half from target samples. The batch-size was set as 128 for both source and target. We report the score after repeating Step 1∼3 (please see Sec 3.2) 20000 times. For our baseline, ENT, we used the same network architecture and the same hyper-parameters as used in our proposed method. In this experiment, SGD with learning rate 1.0 × 10 −3 is used to optimize the parameters. For the finetuning of ResNet101, we set batch-size as 32. Due to the limit of GPU memory, we set it as 24 in finetuning ResNext model. We report the score after 20 epochs training. In order to train MMD model, we use 5 RBF kernels with the following standard deviation parameters: DISPLAYFORM0 We changed the number of the kernels and their parameters, but we could not observe significant performance difference. We report the performance after 5 epochs. We could not see any improvement after the epoch. To train a model BID4 ), we used two-layered domain classification networks. Experimentally, we did not see any improvement when the network architecture is changed. According to the original method BID4 ), learning rate is decreased every iteration. However, in our experiment, we could not see improvement, thus, we fixed learning rate 1.0 × 10 −3. We report the accuracy after 1 epoch. The accuracy dropped significantly after the first epoch. We assume this is due to the large domain difference between synthetic and real images. For our new baseline, ENT, we used the same hyper-parameter as we used for our proposed method. Since the accuracy of ENT drops significantly after around 5 epochs, we report the accuracy after 5 epoch updates. ResBlock3 FORMULA5 FIG9, we show how we integrated the features of each layers. We regard the layers of ResNet50 as generator and rest of the networks, namely convolution and upsampling layers as a critic network. The input images were resized to 512x1024 due to the limit of GPU memory. For the same reason, the batchsize was set to one. In FIG8, we show the example of segmented images by DRN-105. The images are cleanly segmented by our proposed method. In this section, we demonstrate how to apply our method in training a Generative Adversarial Network (GAN) applied to semi-supervised learning. We follow the method proposed by (; BID21), who use a K-class classification network as a critic to train a GAN in the semi-supervised setting. Approach. In contrast to the domain adaptation setting, here G tries to generate images which fool the critic C. Also, in this setting, we are given labeled and unlabeled real images from the same domain. Then, we train the critic to classify labeled images correctly and to move unlabeled images far from the decision boundary. To achieve this, we propose to train the critic with the following objective: SVHN (% errors) CIFAR (% errors) Labeled Only SDGM (Maaløe et al. 16.61 ± 0.24 -CatGAN (Springenberg FORMULA6 Table 4 : Comparison with state-of-the-art methods on two benchmark datasets. Only methods without data augmentation are included. We used the same critic architecture as used in ImpGAN. DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 where X L denotes the subset of labeled samples, X u denotes unlabeled ones and X g denotes images generated by G and H denotes entropy as Eq.6 shows. The critic is trained to minimize the loss on labeled samples in the first term. Since unlabeled images should be far away from the decision boundary and should be distributed uniformly among the classes, we add the second and fourth term. The third term encourages the critic to detect fake images generated near the boundary. The objective of G is as follows, DISPLAYFORM3 where the second term encourages generated images to be similar to real images, which is known to be effective to stabilize the training. The first term encourages the generator to create fake images which should be placed far away from the boundary. Such images should be similar to real images because they are likely to be assigned to some class with high probability. Here, we update C and G same number of times. Experiment. We evaluate our proposed GAN training method by using SVHN and CIFAR10 datasets, using the critic network architecture from BID21). We set the batch size as 100 and used Adam with learning rate 2.0 × 1.0 −4 for optimizer. After the conv6 layer of the critic, we constructed a classifier which was not concerned with adversarial learning process. In the experiment on SVHN, we replaced Weight Normalization with Batch Normalization for C. Also, in the experiment on CIFAR10, we construct a classifier from a middle layer of the critic, which is not incorporated into the adversarial training step. This is motivated by the insight that the critic in our method is trained to be too sensitive to the dropout noise as we explained in Sec 3.3.Results. FIG0, we can see that ADR seems to generate realistic SVHN images. Some images are significantly blurred, but most of the images are clear and diverse. As for generated CIFAR10 images, they do not seem as realistic, but some objects appear in most images. In Table 4, we can see that the accuracy of the critic trained by our method has better performance than other models for SVHN. For CIFAR10, the accuracy was slightly worse than other state-of-the-art methods. We conclude that, despite its clear advantage on the domain adaptation tasks, our method produces mixed on the SSL tasks. It could still be useful for SSL, however, it needs further exploration to improve the accuracy. For example, in Eq. 6, we propose to maximize the entropy of the marginal class distribution of the unlabeled real images, as well as forcing them to be far from the boundary. However, these objectives may contradict each other, which may in turn degrade the performance. In late-breaking , BID1 theoretically showed that just generating fake images that are far from decision boundaries does not help to improve accuracy in training GANs in the setting of SSL. Further improvement of our SSL approach based on these is an interesting direction for future work.

represent the tldr for finding the corresponding scientific article

We present a new adversarial method for adapting neural representations based on a critic that detects non-discriminative features.

Represent the short summary for finding the corresponding scientific article (long)

Operations in the GAN latent space can induce a distribution mismatch compared to the training distribution, and we address this using optimal transport to match the distributions.


Question: represent the article for finding a single-sentence tldr

Question: The use of deep learning for a wide range of data problems has increased the need for understanding and diagnosing these models, and deep learning interpretation techniques have become an essential tool for data analysts. Although numerous model interpretation methods have been proposed in recent years, most of these procedures are based on heuristics with little or no theoretical guarantees. In this work, we propose a statistical framework for saliency estimation for black box computer vision models. We build a model-agnostic estimation procedure that is statistically consistent and passes the saliency checks of. Our method requires solving a linear program, whose solution can be efficiently computed in polynomial time. Through our theoretical analysis, we establish an upper bound on the number of model evaluations needed to recover the region of importance with high probability, and build a new perturbation scheme for estimation of local gradients that is shown to be more efficient than the commonly used random perturbation schemes. Validity of the new method is demonstrated through sensitivity analysis. Deep learning models have achieved great predictive performance in many tasks. However, these complex, often un-tractable models are difficult to interpret and understand. This lack of interpretability is a major barrier for their wide adoption, especially in domains (e.g., medicine) where models need to be qualitatively understood and/or verified for robustness. In order to address these issues, several interpretation approaches have been proposed in the last few years. A group of methods are based on visualizations, either by quantifying the effect of particular neurons or features, or by creating new images that maximize the target score for specific classes (; ;). A large collection of the techniques build saliency maps by attributing the gradients of the neural network to the input image through various procedures or by finding perturbations that significantly change the output (; ; ; ; ; ; ; ; a; ;). Another class of approaches treat the deep learner as a black-box. In this domain, use a Parzen window classifier to approximate the target classifier locally. propose the LIME procedure, where small perturbations on the instance are used to obtain additional samples with which a sparse linear model is fit. propose SHapley Additive exPlanation(SHAP), which combines the Shapley value from the game theory with the additive feature attribution methods. They also make connections of the SHAP procedure with various existing methods including LRP, LIME and DeepLIFT. propose L-and C-Shapley procedures which can reliably approximate the Shapley values in linear time with respect to the number of features. Majority of the listed methods are heuristics which are constructed according to certain desirable qualities. For these methods, it is not clear what the main estimand is, if it can be consistently estimated or if (and how) the estimand can be computed more efficiently. In fact, according to the recent research by Adebayo et al. (2018b), most methods with great visual inspection lack sensitivity to the model and the data generating process. Theoretical explanation for why guided back-propagation and deconvolutional methods perform image recovery is provided by. In this work, we propose a statistically valid technique for model-agnostic saliency estimation, and prove its consistency under reasonable assumptions. Furthermore, our method passes the sanity checks given by Adebayo et al. (2018b). Through our analysis, we obtain insights into how to improve the accuracy and reliability of our approach. We note that there is recent work by where they provide a saliency estimation technique with theoretical guarantees -more specifically, FDR control. Although their procedure is very promising from a statistical perspective, and theoretically valid under a very general set of assumptions, their technique requires human input and has a significant computational load as it uses a generative model for filling in certain regions of the target image. Our main contributions are as follows: • We introduce a new saliency estimation framework for CNNs and propose a new method based on input perturbation. Our procedure requires solving a linear program, and hence the estimates can be computed very efficiently. Furthermore, the optimization problem can be recast as a "parametric simplex" , which allows the computation of the full solution path in an expedient manner. • We establish conditions under which the significant pixels in the input can be identified with high probability. We present finite-sample convergence rates that can be used to determine the number of necessary model evaluations. • We find that the noise distribution for the perturbation has a substantial effect on the convergence rate. We propose a new perturbation scheme which uses a highly correlated Gaussian, instead of the widely used independent Gaussian distribution. In the following section, we define the linearly estimated gradient (LEG), which is the saliency parameter of interest (i.e. the estimand), and introduce our statistical framework. In section 3, we propose a regularized estimation procedure for LEG that penalizes the anisotropic total-variation. We provide our theoretical in Section 4 and the of our numerical comparisons in Section 5. For a matrix B, we use vec(B) and vec −1 (B) to denote its vectorization and inverse vectorization, respectively. The transpose of a matrix B is given by B T and we use B + for its pseudo-inverse. The largest and smallest eigenvalue of a symmetric matrix B are denoted by λ max (B) and λ min (B). For a set S, we use S C to denote its complement. For a vector u ∈ R p and a set S ⊆ [1, . . ., p], we use u S to refer to its components indexed by elements in S. The q-norm for a vector u is given by u q and we use B F r for the Frobenius norm of a matrix B. The vector of size p whose values are all equal to 1 is denoted by 1 p. Similarly, we use 1 p1×p2 and 0 p1×p2 to denote a p 1 × p 2 matrix whose entries are equal to 1 and 0, respectively. Finally, for a continuous distribution F, we use F + x 0 to denote a distribution that is mean-shifted by x 0, i.e. F (z) = G(z − x 0) for all z, where In gradient based saliency approaches, the main goal is to recover the gradient of the deep learner with respect to the input. More specifically, let f (x) be a deep learner, f: X →, where X is the input space, e.g., 28×28 for the MNIST dataset, where the input are given as 28 by 28 sized images. In this notation, the output is the probability of a specific class, for instance P model (x is a 9); although this can be modified to check for comparative quantities by setting the output as f (x) = f 9 (x) − f 7 (x) = P model (x is a 9) − P model (x is a 7). Then, local saliency is defined as the derivative of f (·) with respect to the input, evaluated at a point of interest x 0 ∈ X, i.e. ∇ x f (x)| x=x0. However, in practice, local saliency is often too noisy and one instead uses an average of the gradient around x 0 . In order to study the saliency procedure from a statistical perspective, we start by defining an estimand, whose definition is motivated by the LIME procedure . Definition 1 (LEG). For a continuous distribution F, an initial point x 0 ∈ X with X ⊂ R p1×p2, and a function f: X → [−1, 1], the linearly estimated gradient (LEG), γ ∈ R p1×p2 is given by LEG is based on a first order Taylor series expansion of the function f (x) around the point of interest x 0. The estimand is a proxy for the local gradient, and is the coefficient that gives the best linear approximation, in terms of the squared error, among all possible choices. The distribution F determines the range of points the analyst wants to consider. We visually demonstrate LEG on two toy examples with a single pixel (i.e. p 1 = p 2 = 1) in Figure 1. Figure 1a, we compare LEG to the gradient, which is very localized. If f (x) is a highly varying function, then the gradient is too noisy, and the saliency score provided by LEG is more meaningful. In Figure 1b, we show LEG for two different distributions. For the distribution with the larger variance, LEG evaluates the input's effect on the output for a larger neighborhood around x 0. We note that the variance of F has a large effect on LEG. As F converges to a point mass at 0, if f (x) is twice continuously differentiable in the neighborhood of x 0, then γ → ∇ x f (x). On the other hand, if F has high variance, then samples from x 0 + F are substantially different from x 0 and LEG might no longer be useful for interpreting the model at x 0. This phenomenon can also described in terms of local vs global interpretation: for F with a small variance, LEG provides a very local interpretation, i.e. a gradient that is valid in a small neighborhood around x 0, and as the variance of F increases, LEG produces a more global interpretation, since a larger neighborhood around x 0 is considered in the calculation. LEG has an analytical solution as the next lemma shows. Lemma 1. Let Z be the random variable with a centered distribution F, i.e. Z ∼ F and E[Z] = 0 p1×p2. Assume that covariance of vec(Z) exists, and is positive-definite. Proof of the lemma is provided in the Appendix. Lemma 1 shows that the LEG can be written as an affine transformation of a high dimensional integral where the integrand is (f ( This analysis also suggests an empirical estimate for the LEG, by replacing the expectation with the empirical mean. The empirical mean can be obtained by sampling x from F + x 0, calculating f (x), and then applying Lemma 1. More formally, let x 1,..., x n be random samples from F + x 0, and let y 1,..., y n be the function evaluations with As the function f (x) is bounded and F has a positive-definite covariance matrix, then it follows that as n → ∞,γ → γ. However, classical linear model theory shows that rate of the convergence is very slow, on the order of 1 λmin(Σ) p 1 p 2 /n, where p 1 and p 2 are the dimensions of X. This severely limits the practicality of the empirical approach. In the next section we propose to use regularization in order to obtain faster convergence rates. For interpretation of image classifiers, one expects that the saliency scores are located at a certain region, i.e. a contiguous body or a union of such bodies. This idea has lead to various procedures that estimate saliency scores by penalizing the local differences of the solution, often utilizing some form of the total variation (TV) penalty . The approach is very sensible from a practical point of view: Firstly, it produces estimates that are easy to interpret as the important regions can be easily identified; secondly, penalization significantly shrinks the variance of the estimate and helps produce reliable solutions with less model evaluations. In the light of the above, we propose to estimate the LEG coefficient with an anisotropic L 1 TV penalty. For a hyperparameter, L ≥ 0, the TV-penalized LEG estimate is given as γ = vec −1 (g) where g is the solution of the following linear program where D ∈ R (2p1p2−p1−p2)×(p1p2) is the differencing matrix with D i,j = 1, D i,k = −1 if the j th and the k th component of g are connected on the two dimensional grid. Our method is based on the "high confidence set" approach which has been successful in numerous applications in high dimensional statistics (; ;). The set of g that satisfy the constraint in the formulation is our high confidence set; if L is chosen properly, this set contains the true LEG coefficient, γ(f, x 0, F), with high probability 1. This setup ensures that the distance between γ andγ is small. When combined with the TV penalty in the objective function, the procedure seeks to find a solution that both belongs to the confidence set and has sparse differences on the grid. Thus, the estimator is extremely effective at recovering γ that have small total variation. The proposed method enjoys low computational complexity. The problem in equation 4 is a linear program and can be solved in polynomial time, for instance by using a primal-dual interior-point method for which the time complexity is O (p 1 p 2) 3.5 . However, in practice, solutions can be obtained much faster using simplex solvers. In our implementations, we use MOSEK, a commercial grade simplex solver by , and are able to obtain a solution in less than 3 seconds on a standard 8-core PC for a problem of size p 1 = p 2 = 28. Additionally, the alternative formulation (provided in the Appendix) can be solved using parametric simplex approaches which yield the whole solution path in L . The last point is often a necessity in deployment when L needs to be tuned according to some criteria. We note that the procedure does not require any knowledge about the underlying neural network and is completely model-agnostic. In fact, in applications where security or privacy could be a concern and returning multiple prediction values needs to be avoided, the term given by n i=1 vec (ỹ i z i) can be computed on the side and supplied alongside the prediction. In Figure 2, we show the ing estimates of the method with n = 500 model evaluations for a VGG-19 network. For the distribution F, we use a multivariate Gaussian distribution with the proposed perturbation scheme in Section 4.2. We computeγ separately for each channel, and then sum the absolute values of the different channels to obtain the final saliency score. In this section, we analyze the procedure from a theoretical perspective and derive finite sample convergence rates of the proposed LEG-TV estimator. As we noted earlier, this analysis also gives us insight on the properties of the ideal perturbation distribution. We first present our condition, which has a major role in the convergence rate of our estimator. The condition is akin to the restricted eigenvalue condition with adjustments specific to our problem. Assumption 1. Let D + be the pseudo-inverse of the differencing matrix D, and denote the elements of singular value decomposition of D as U, Θ, V where D = U ΘV T. Furthermore, denote the last p 1 p 2 − p 1 − p 2 columns of U that correspond to zero singular values as U 2. For the covariance matrix Σ, and any set S with size s, it holds that κ > 0, where The following theorem is our main ., where Z ∼ F and E[Z] = 0 p1×p2. Letγ be the LEG-TV estimate with L = 2 D + 1 log (p 1 p 2 /) /n. If Assumption 1 holds for the covariance matrix Σ with constant κ, then with probability 1 −, where m ∈ R is a mean shift parameter, s is the number of non-zero elements in Dγ The proof is built on top of the "high confidence set" approach of. In the proof, we first establish that, for an appropriately chosen value of L, γ * = γ(f, x 0, F) satisfies the constraint in equation 4 with high probability. Then, we make use of TV sparsity ofγ and γ * to argue that the two quantities cannot be too far away from each other, since both are in the constraint set. The full proof is provided in the Appendix. Our theorem has two major implications: 1. We can recover the true parameter as the number of model evaluations increase. That is, TV penalized LEG is a statistically consistent model interpretation scheme. Furthermore, our states that, ignoring the log terms, one needs n = O(s (p 1 p 2) 1/2 ) many model evaluations to reliably recover γ *. 2. Our bound depends on the constant κ, which further depends on the choice of Σ for the perturbation scheme. It is possible to obtain faster rates of convergence with a carefully tuned choice of Σ. As a side note, since γ * also depends on Σ, the estimand changes when Σ is adjusted. In other words, our states that certain estimands require less samples. We note that our procedure identifies the LEG coefficient up to a mean shift parameter, m, which is the average of the true LEG coefficient γ. In practice, the average can be consistently estimated (for instance, using the empirical version of LEG in equation 3), and the mean can be subtracted to yield consistent estimates for γ. However, in our numerical studies, we see that this mean shift is almost non-existent: LEG-TV yields solutions that has no mean differences with the LEG coefficient, which we define as the solution of the empirical version as n → ∞. In our main , we established that the convergence of our estimator depends on the quantity κ which is related to the spectral properties of Σ. In this subsection we explore the ramifications of the assumption. Our main in Theorem 1 states that the rate of convergence to the true LEG coefficient is inversely proportional to the term κ. Thus, perturbation schemes for which the restricted eigenvalues are large, as defined in Definition 1, yield saliency maps that require less samples to estimate the LEG. We note that most of the saliency estimation procedures that make use of perturbations take these perturbations to be independent, which in a covariance matrix that is equal to the identity matrix, Σ = σ 2 I (p1p2)×(p1p2) for some σ 2 > 0. For LEG estimation without penalization, i.e. using equation 1, this choice is also optimal as the convergence rates under the normal setup depend on 1/λ min (Σ). However, when one seeks to find an estimate for which the solution is sparse in the TV norm, this choice is no longer ideal as demonstrated by our theorem. In order to choose the covariance matrix of our perturbation scheme in a manner that maximizes the bound in equation 5, one also needs some prior information about the size of S, s. As that requires estimation of s, and a complex optimization procedure, we instead propose a heuristic: we choose Σ so that its eigenvectors match D + ∆ for vectors ∆ with unit-norm and U T 2 ∆ = 0. This choice fixes p 1 p 2 − 1 many of the eigenvectors of Σ. For the last eigenvector, we use the one vector as it is orthogonal to the rest of the eigenvectors. Our proposed perturbation scheme is as follows: 1. Compute the singular value decomposition of D, and let D = U ΘV T. for some choice of σ 2 > 0. with the proposed Σ, the numerator in equation 5 reduces to σ 2 ∆ T ∆ and hence κ = σ 2. Without any additional assumptions on S, this is the maximal value for κ. Figure 3: Selected eigenvectors of the proposed Σ. The eigenvectors, which contain the principal directions of the distribution, have maxima and minima in adjacent locations. Distributions drawn with these properties perform as object detectors as they can be used to detect existence (or nonexistence) of significant pixels at these locations. We plot some of the eigenvectors for our proposed Σ with p 1 = p 2 = 28 in Figure 3. These eigenvectors are the principal directions of the perturbation distribution F, and the samples drawn from F contain a combination of these directions. We see these samples will have sharp contrasts at certain locations. This is very intuitive: The perturbation scheme is created for a specific problem where boundaries for objects are assumed to exist, and large jumps in the magnitude of the distribution help our method recover these boundaries efficiently. We conclude this section with a demonstration of the perturbation scheme using Gaussian noise. In Figure 4, we plot a digit from the MNIST dataset , along with instances obtained by independent perturbation and by our suggested distribution. LEG-TV procedure has two tuning parameters: (i) F, which determines the structure of the perturbation; and (ii) L, which controls the sparsity of the chosen interpretation. Regarding F, we propose to use a multivariate Gaussian distribution as it is easy to sample from. For Σ, we propose a theoretically driven heuristic for determining the correlation structure of Σ in Section 4.2. However, the choice of the magnitude of Σ, i.e. σ 2, is left to the user. If this quantity is chosen too low, then the added perturbations are small in magnitude, and the predictions of the neural network do not change, ing in a LEG near zero. On the other hand, with a very large value of σ 2, the have too much variance as some of the pixel values are set to the minimum or the maximum pixel intensity. In our implementations, we find that setting σ 2 to be between 0.05 and 0.30 in reasonable solutions. We determine this range by computing perturbations of various sizes on numerous images using the VGG-19 classifier. The provided range is found to create perturbations large enough to change the prediction probabilities but small enough to avoid major changes in the image. Most of our presented are given for σ 2 = 0.10. For the choice of L, we propose two solutions: The first is the theoretically suggested quantity given in Theorem 1, although this often in estimates that are too conservative. Our second method is a heuristic based on some of the quantities in the optimization problem and we use this for our demonstrations. We set L = K L L max where K is a constant between 0 and 1 and L max is the smallest value of L for which the solution in equation 4 would with g = 0; i.e.. We use K L = 0.05 or K L = 0.10 in our implementations. We note that is possible to obtain the solution for all L by using a parametric simplex solver , or by starting with a large initial L, then using the solution of the program as a warm-start for a smaller choice of L. Both approaches return the solution path for all L, and might be more desirable in practice than relying on heuristics. In this section, we demonstrate the robustness and validity of our procedure by two numerical experiments. In Section 5.1, we perform sanity checks as laid out by Adebayo et al. (2018b), and show that the LEG-TV estimator fails to detect objects when the weights of the neural network are chosen randomly. In Section 5.2, we implement a sensitivity analysis in which we use various saliency methods to compute regions of importance, and then perturb these regions in order to see their effect on the prediction. For the deep learner, we use VGG-19 . For computational efficiency, we compute saliency maps on a 28 by 28 grid (i.e.γ ∈ R 28×28) although the standard input for VGG-19 is 224 by 224. The perturbations on the image are scaled up by 8 via upsampling in order for the dimensions to match. In Adebayo et al. (2018b), the validity of saliency estimation procedures are tested by varying the weights of the neural network. In a technique named, "cascading randomization", authors propose to replace the fitted weights of a CNN layer by layer, and compute the saliency scores with each change. As a deep learner with randomly chosen weights should have no prediction power, one expects to see the same effect in the ing saliency scores: namely, as more of the weights are perturbed, the explanation offered by interpretability methods should become more and more meaningless. Surprisingly, Adebayo et al. (2018b) show that most commonly adopted interpretation procedures provide some saliency even after full randomization, and conclude that these methods act as edge detectors. Our procedure treats the classifier as a black-box and the explanations offered by LEG-TV are based solely on the predictions made by the neural network. During the sanity check, when the weights of the neural network are randomly perturbed, the predictions change significantly and no longer depend on the input. Thus, we expect the local linear approximations of the underlying function to be flat, which would in saliency scores of zero for all of the pixels. Finally, small artifacts that might arise in this process, such as positive or negative saliency scores with no spatial structure, should be smoothed over due to the TV penalty, further robustifying our procedure. In order to verify our intuition, we perform cascading randomization on the weights of a VGG-19 network. For all of the images in our analysis, we find that the LEG-TV estimate,γ, is reduced to zero after randomization of either the top (i.e. logits) or the second top layer (i.e. second fully connected layer). The of our experiment for two images are given in Figure 5. It is seen that after the weights are perturbed, the LEG-TV method fails to detect any signal that could be used for interpretation. In fact, due to penalization, the estimate is set to zero. These show that the interpretation given by our proposed method is reliable and is dependent on the classifier....... Figure 5: Results of the sanity check with cascading randomization. The network weights are replaced by random numbers in a cascading order, starting from the last layer. LEG is equal to zero for all pixel values immediately after the first randomization. For our second validity test, we use various interpretation models to compute regions of high importance. We then mask these regions by decreasing the value of the pixels to zero which is equivalent to painting them black. We compute and assess the difference of the predictions for the target class with each perturbation. We compare our method against four alternatives: GradCAM , LIME , SHAP and C-Shapley . The last three methods are chosen as they are model-agnostic, like LEG, and do not make use of the architecture of the neural network. GradCAM is chosen due to its popularity. The saliency maps using C-Shapley and LEG-TV are computed for a 28 by 28 grid. In order to make the comparison between the methods more fair, we downsize the saliency maps ing from GradCAM, LIME and SHAP to the same size. Interestingly, we find that this step improves the performance of these estimators; that is, the perturbations identified using the low resolution saliency maps in faster drops in the predicted score. For LEG-TV, LIME and SHAP, the saliency scores are computed using 3000 model evaluations, where as C-Shapley requires 3136 (28×28×4) evaluations. For LEG-TV, we provide two solutions, a sparse solution which corresponds to a larger choice of the penalty parameter L and a noisy solution which is obtained with a smaller choice of L, denoted by LEG and LEG0, respectively. We present the for 500 images that are randomly chosen from a subsample of the ImageNet dataset 2. The average of the log odds ratios across the 500 images are provided in Figure 6. We see that as the size of the perturbation increases, the predictions for the target class drop for all of the methods. The slope is sharpest for SHAP and LEG0, suggesting that these two methods identify pixels that are crucial for the predictions. Figure 6: Results of sensitivity analysis. Log of the predicted probability for the target class is plotted versus the size of the perturbation. The locations for the perturbations are determined by the saliency procedures. Predictions should decrease at a fast rate for interpretability methods that can reliably identify regions of importance. In that regard, SHAP and LEG0 appear to be the most accurate in determining the critical pixels, followed by LEG, GradCAM, C-Shapley and LIME. In Figure 7, we plot the top 10% most salient pixels according to different procedures for three images in the dataset. The pixels chosen by SHAP appear to correspond to specific a convolution pattern and the chosen region is not contiguous. On the other hand, pixels identified by LEG-TV are visually meaningful to the human eye and contain pixels that are more likely to be relevant for the prediction. LEG-TV selects different parts of the crane in the first image, and the face of the Pekinese dog in the second. In the last image, where a soap dispenser is misclassified as a soda bottle, LEG-TV relates the classification to the label and the barcode of the bottle -parts that are often seen on soda bottles. For the same image, LEG-TV also selects the fixtures in the , which could have been mistaken by the classifier as the cap of the soda bottle. We have proposed a statistical framework for saliency estimation that relies on local linear approximations. Utilizing the new framework, we have built a computationally efficient saliency estimator that has theoretical guarantees. Using our theoretical analysis, we have identified how the sample complexity of the estimator can be improved by altering the model evaluation scheme. Finally, we have shown through empirical studies that (i) unlike most of its competitors, our method passes the recently proposed sanity checks for saliency estimation; and (ii) pixels identified through our approach are highly relevant for the predictions, and our method often chooses regions with higher saliency compared to regions suggested by its alternatives. Our linear program can also be recast by a change of variables and setting α = Dg. In this case, the elements of α correspond to differences between adjoint pixels. This program can be written as: + is the pseudo-inverse of D and U 2 is related to the left singular vectors of D. More precisely, letting D = U ΘV T denote the singular value decomposition of D, U 2 is the submatrix that corresponds to the columns of U for which Θ j is zero. The linearity constraint ensures that the differences between the adjoint pixels is proper. Derivation of the alternative formulation follows from Theorem 1 in and is omitted. This formulation can be expressed in the standard augmented form, i.e. min Ax=b,x≥0 c T x, by writ- where y = 1 n n i=1f (x i)x i and m = 2p 1 p 2 −p 1 −p 2. The γ coefficient in the original formulation can be obtained by setting A.2 PROOF OF THEOREM 1 Our proof depends on the following lemma. Lemma 2. For L ≥ 2 D + 1 log (p 1 p 2 /) /n, γ * is in the feasibility set with probability 1 −, that is Proof. For ease of notation, let We also assume that the images have been rescaled so that the maximum value ofx i is 1 (without rescaling, the maximum would be given as the largest intensity, i.e. 255). Since, the function values are also in the range given by [-2,2], we can bound |z i,j |, that is Under review as a conference paper at ICLR 2020 The proof follows by applying the McDiarmid's inequality for each row of the difference and then taking the supremum over the terms. By application of McDiarmid's inequality, we have that Let L = 2 D + 1 log (p 1 p 2 /2) /n. Then, taking a union bound over all variables, we have Now note that that the feasibility set for any L ≥ L contains that of L and thus γ * is automatically included. We now present the proof of the theorem. Note that the technique is based on the Confidence Set approach by. In the proof, we use γ to refer to vec(γ) for ease of presentation. Proof. First, let the high probability set for which Lemma 2 holds by A. All of the following statements hold true for A. We let ∆ = D (γ − γ *). We know that Dγ 1 ≤ Dγ * 1 since both are in the feasibility set, as stated in Lemma 2. Let α * = Dγ *,α = Dγ and define S = {j : α * j = 0}, and the complement of S as S C. By assumption of the Theorem, we have that the cardinality of S is s, i.e. |S| = s. Now let ∆ S as the elements of ∆ in S. Then, using the above statement, one can show that ∆ S 1 ≥ ∆ S C 1. Note, and ∆ S 1 ≥ ∆ S C 1 follows immediately. Furthermore where the last line uses the previous . Additionally, note that where the first inequality follows by Holder's inequality and the second follows from Lemma 2 and the fact that bothγ and γ * are in the feasibility set for L = 2 D + 1 log (p 1 p 2 /) /n. We further bound the right hand side of the inequality by using the previous , which gives Next, we bound ∆ 2 by combining the previous . Now, by assumption of the Theorem, we have that Dividing both sides by ∆ 2, we obtain that

Represent the short summary for finding the corresponding scientific article (long).

We propose a statistical framework and a theoretically consistent procedure for saliency estimation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.