Question: Represent the article for finding a single-sentence TLDR!

Question: When an image classifier makes a prediction, which parts of the image are relevant and why? We can rephrase this question to ask: which parts of the image, if they were not seen by the classifier, would most change its decision? Producing an answer requires marginalizing over images that could have been seen but weren't. We can sample plausible image in-fills by conditioning a generative model on the rest of the image. We then optimize to find the image regions that most change the classifier's decision after in-fill. Our approach contrasts with ad-hoc in-filling approaches, such as blurring or injecting noise, which generate inputs far from the data distribution, and ignore informative relationships between different parts of the image. Our method produces more compact and relevant saliency maps, with fewer artifacts compared to previous methods. The decisions of powerful image classifiers are difficult to interpret. Saliency maps are a tool for interpreting differentiable classifiers that, given a particular input example and output class, computes the sensitivity of the classification with respect to each input dimension. BID3 and BID2 cast saliency computation an optimization problem informally described by the following question: which inputs, when replaced by an uninformative reference value, maximally change the classifier output? Because these methods use heuristic reference values, e.g. blurred input BID3 or random colors BID2, they ignore the context of the surrounding pixels, often producing unnatural in-filled images (Figure 2). If we think of a saliency map as interrogating the neural network classifier, these approaches have to deal with a somewhat unusual question of how the classifier responds to images outside of its training distribution. To encourage explanations that are consistent with the data distribution, we modify the question at hand: which region, when replaced by plausible alternative values, would maximally change classifier output? In this paper we provide a new model-agnostic framework for computing and visualizing feature importance of any differentiable classifier, based on variational Bernoulli dropout BID4. We marginalize out the masked region, conditioning the generative model on the non-masked parts of the image to sample counterfactual inputs that either change or preserve classifier behavior. By leveraging a powerful in-filling conditional generative model we produce saliency maps on ImageNet that identify relevant and concentrated pixels better than existing methods. Gradient-based approaches BID12 BID13 BID17 BID10 ) derive a saliency map for a given input example and class target by computing the gradient of the classifier output with respect to each component (e.g., pixel) of the input. The reliance on the local gradient information induces a bias due to gradient saturation or discontinuity in the DNN activations BID11. BID0 observed that some gradientbased saliency computation reflect an inductive bias due to the convolutional architecture, which is independent of the network parameter values. To explain its response to a particular input x we partition the input x into masked (unobserved) region x r and their complement x = x r ∪ x \r. Then we replace the x r with uninformative reference valuê x r to test which region x r is important for classifier's output p M (c|x r, x \r). Heuristic in-filling BID3 computesx r ad-hoc such as image blur. This biases the explanation when samples [x r, x \r] deviate from the data distribution p(x r, x \r). (1c) We instead sample x r efficiently from a conditional generative model x r ∼ p G (x r |x \r) that respects the data distribution. Reference-based approaches analyze the sensitivity of classifier outputs to the substitution of certain inputs/pixels with an uninformative reference value. BID11 linearly approximates this change in classifier output using an algorithm resembling backpropagation. This method is efficient and addresses gradient discontinuity, but ignores nonlinear interactions between inputs. BID1 optimizes a variational bound on the mutual information between a subset of inputs and the target, using a variational family that sets input features outside the chosen subset to zero. In both cases, the choice of value as reference limits applicability to simple image domains with static like MNIST. BID18 computes the saliency of a pixel (or image patch) by treating it as unobserved and marginalizing it out, then measuring the change in classification outcome. This approach is similar in spirit to ours. The key difference is that where BID18 iteratively execute this computation for each region, we leverage a variational Bernoulli distribution to efficiently search for optimal solution while encouraging sparsity. This reduces computational complexity and allows us to model the interaction between disjoint regions of input space. BID3 computes saliency by optimizing the change in classifier outputs with respect to a perturbed input, expressed as the pixel-wise convex combination of the original input with a reference image. They offer three heuristics for choosing the reference: mean input pixel value (typically gray), Gaussian noise, and blurred input. BID2 amortize the cost of estimating these perturbations by training an auxiliary neural network.3 propose two objectives for computing the saliency map:• Smallest Deletion Region (SDR) considers a saliency map as an answer to the question: What is the smallest input region that could be removed and swapped with alternative reference values in order to minimize the classification score? • Smallest Supporting Region (SSR) instead poses the question: What is the smallest input region that could substituted into a fixed reference input in order to maximize the classification score?Solving these optimization problems (which we formalize below) involves a search over input masks, and necessitates reference values to be substituted inside (SDR) or outside (SSR) the masked region. These values were previously chosen heuristically, e.g., mean pixel value per channel. We instead consider inputs inside (SDR) or outside (SSR) the masked region as unobserved variables to be marginalized efficiently by sampling from a strong conditional generative model 1. We describe our approach for an image application where the input comprises pixels, but our method is more broadly applicable to any domain where the classifier is differentiable. Generative Methods Consider an input image x comprising U pixels, a class c, and a classifier with output distribution p M (c|x). Denote by r a subset of the input pixels that implies a partition of the input x = x r ∪ x \r. We refer to r as a region, although it may be disjoint. We are interested in the classifier output when x r are unobserved, which can be expressed by marginalization as We then approximate p(x r |x \r) by some generative model with distribution p G (x r |x \r) (specific implementations are discussed in section 4.1). Then given a binary mask 2 z ∈ {0, 1} U and the original image x, we define an infilling function 3 φ as a convex mixture of the input and reference with binary weights, DISPLAYFORM0 The classification score function s M (c) represents a score of classifier confidence on class c; in our experiments we use log-odds: DISPLAYFORM0 SDR seeks a mask z yielding low classification score when a small number of reference pixels are mixed into the mask regions. Without loss of generality 4, we can specify a parameterized distribution Given an input, FIDO-CA finds a minimal pixel region that preserves the classifier score following in-fill by CA-GAN BID16. BID2 (Realtime) assigns saliency coarsely around the central object, and the heuristic infill reduces the classifier score. We mask further regions (head and body) of the FIDO-CA saliency map by hand, and observe a drop in the infilled classifier score. The label for this image is "goose".over masks q θ (z) and optimize its parameters. The SDR problem is a minimization w.r.t θ of DISPLAYFORM1 On the other hand, SSR aims to find a masked region that maximizes classification score while penalizing the size of the mask. For sign consistency with the previous problem, we express this as a minimization w.r.t θ of DISPLAYFORM2 Naively searching over all possible z is exponentially costly in the number of pixels U. Therefore we specify q θ (z) as a factorized Bernoulli: DISPLAYFORM3 Bern(z u |θ u).This corresponds to applying Bernoulli dropout BID14 to the input pixels and optimizing the per-pixel dropout rate. θ is our saliency map since it has the same dimensionality as the input and provides a probability of each pixel being marginalized (SDR) or retained (SSR) prior to classification. We call our method FIDO because it uses a strong generative model (see section 4.1) to Fill-In the DropOut region. To optimize the θ through the discrete random mask z, we follow in computing biased gradients via the Concrete distribution BID9 BID7; we use temperature 0.1. We initialize all our dropout rates θ to 0.5 since we find it increases the convergence speed and avoids trivial solutions. We optimize using Adam BID8 with learning rate 0.05 and linearly decay the learning rate for 300 batches in all our experiments. Our PyTorch implementation takes about one minute on a single GPU to finish one image. compute saliency by directly optimizing the continuous mask z ∈ U under the SDR objective, withx chosen heuristically; we call this approach Black Box Meaningful Perturbations (BBMP). We instead optimize the parameters of a Bernoulli dropout distribution q θ (z), which enables us to sample reference valuesx from a learned generative model. Our method uses mini-batches of samples z ∼ q θ (z) to efficiently explore the huge space of binary masks and obtain uncertainty estimates, whereas BBMP is limited to a local search around the current point estimate of the mask z. See Figure 5 for a pseudo code comparison. In Appendix A.1 we investigate how the choice of algorithm affects the ing saliency maps. To avoid unnatural artifacts in φ(x, z), BID3 and BID2 additionally included two forms of regularization: upsampling and total variation penalization. DISPLAYFORM0 With φ, compute L by Equation 4 or 5 Update z with ∇ z L end while Return z as per-feature saliency map DISPLAYFORM1 With φ, compute L by Equation FORMULA3 Upsampling is used to optimize a coarser θ (e.g. 56 × 56 pixels), which is upsampled to the full dimensionality (e.g. 224 × 224) using bilinear interpolation. Total variation penalty smoothes θ by a 2 regularization penalty between spatially adjacent θ u. To avoid losing too much signal from regularization, we use upsampling size 56 and total variation as 0.01 unless otherwise mentioned. We examine the individual effects of these regularization terms in Appendices A.2 and A.4, respectively. We first evaluate the various infilling strategies and objective functions for FIDO. We then compare explanations under several classifier architectures. In section 4.5 we show that FIDO saliency maps outperform BBMP BID3 in a successive pixel removal task where pixels are in-filled by a generative model (instead of set to the heuristic value). FIDO also outperforms the method from BID2 on the so-called Saliency Metric on ImageNet. Appendices A.1-A.6 provide further analysis, including consistency and the effects of additional regularization. We describe several methods for producing the reference valuex. The heuristics do not depend on z and are from the literature. The proposed generative approaches, which producex by conditioning on the non-masked inputs x z=0, are novel to saliency computation. Heuristics: Mean sets each pixel ofx according to its per-channel mean across the training data. Blur generatesx by blurring x with Gaussian kernel (σ = 10) BID3. Random samplesx from independent per-pixel per-channel uniform color with Gaussians (σ = 0.2). Generative Models: Local computesx as the average value of the surrounding non-dropped-out pixels x z=0 (we use a 15 × 15 window). VAE is an image completion Variational Autoencoder BID6. Using the predictive mean of the decoder network worked better than sampling. CA is the Contextual Attention GAN BID16; we use the authors' pre-trained model. Here we examine the choice of objective function between L SDR and L SSR; see Figure 6. We observed more artifacts in the L SDR saliency maps, especially when a weak in-filling method (Mean) is used. We suspect this unsatisfactory behavior is due to the relative ease of optimizing L SDR. There are many degrees of freedom in input space that can increase the probability of any of the 999 classes besides c; this property is exploited when creating adversarial examples BID15 ). Since Figure 6: Choice of objective between L SDR and L SSR. The classifier (ResNet) gives correct predictions for all the images. We show the L SDR and L SSR saliency maps under 2 infilling methods: Mean and CA. Here the red means important and blue means non-important. We find that L SDR is more susceptible to artifacts in the ing saliency maps than L SSR.it is more difficult to infill unobserved pixels that increase the probability of a particular class c, we believe L SSR encourages FIDO to find explanations more consistent with the classifier's training distribution. It is also possible that texture is easier for a conditional generative model to fit. To mitigate the effect of artifacts, we use L SSR for the remaining experiments. Figure 7: Comparison of saliency map under different infilling methods by FIDO SSR using ResNet. Heuristics baselines (Mean, Blur and Random) tend to produce more artifacts, while generative approaches (Local, VAE, CA) produce more focused explanations on the targets. Here we demonstrate the merits of using strong generative model that produces substantially fewer artifacts and a more concentrated saliency map. In Figure 7 we generate saliency maps of different infilling techniques by interpreting ResNet using L SSR with sparsity penalty λ = 10 −3. We observed a susceptibility of the heuristic in-filling methods (Mean, Blur, Random) to artifacts in the ing saliency maps, which may fool edge filters in the low level of the network. The use of generative in-filling (Local, VAE, CA) tends to mitigate this effect; we believe they encourage in-filled images to lie closer to the natural image manifold. To quantify the artifacts in the saliency maps by a proxy: the proportion of the MAP configuration (θ > 0.5) that lies outside of the ground truth bounding box. FIDO-CA produces the fewest artifacts by this metric FIG5 ). We use FIDO-CA to compute saliency of the same image under three classifier architectures: AlexNet, VGG and ResNet; see FIG6. Each architecture correctly classified all the examples. We observed a qualitative difference in the how the classifiers prioritize different input regions (according to the saliency maps). For example in the last image, we can see AlexNet focuses more on the body region of the bird, while Vgg and ResNet focus more on the head features. We follow BID3 and BID11 in measuring the classifier's sensitivity to successively altering pixels in order of their saliency scores. Intuitively, the "best" saliency map should compactly identify relevant pixels, so that the predictions are changed with a minimum number of altered pixels. Whereas previous works flipped salient pixel values or set them to zero, we note that this moves the classifier inputs out of distribution. We instead dropout pixels in saliency order and infill their values with our strongest generative model, CA-GAN. To make the log-odds score suppression comparable between images, we normalize per-image by the final log-odds suppression score (all pixels infilled). In FIG0 we evaluate on ResNet and carry out our scoring procedure on 1, 533 randomly-selected correctly-predicted ImageNet validation images, and report the number of pixels required to reduce the normalized log-odds score by a given percent. We evaluate FIDO under various in-filling strategies as well as BBMP with Blur and Random in-filling strategies. We put both algorithms on equal footing by using λ = 1e−3 for FIDO and BBMP (see Section A.1 for further comparisons). We find that strong generative infilling (VAE and CA) yields more parsimonious saliency maps, which is consistent with our qualitative comparisons. FIDO-CA can achieve a given normalized log-odds score suppression using fewer pixels than competing methods. While FIDO-CA may be better adapted to evaluation using CA-GAN, we note that other generative in-filling approaches (FIDO-Local and FIDO-VAE) still out-perform heuristic in-filling when evaluated with CA-CAN.We compare our algorithm to several strong baselines on two established metrics. We first evaluate whether the FIDO saliency map can solve weakly supervised localization (WSL) BID2. After thresholding the saliency map θ above 0.5, we compute the smallest bounding box containing all salient pixels. This prediction is "correct" if it has intersection-over-union (IoU) ratio over 0.5 with any of the ground truth bounding boxes. Using FIDO with various infilling methods, we report the average error rate across all 50, 000 validation images in Table 1. We evaluate the authors' pre-trained model of BID2 5, denoted as "Realtime" in the . We also include five baselines: Max (entire input as the bounding box), Center (centered bounding box occupying half the image), Grad BID12, Deconvnet , and GradCAM BID10. We follow the procedure of mean thresholding in BID3: we normalize the heatmap between 0 and 1 and binarize by threshold θ = αµ i where µ i is the average heatmap for image i. Then we take the smallest bounding box that encompasses all the binarized heatmap. We search α between 0 to 5 with 0.2 step size on a holdout set to get minimun WSL error. The best α are 1.2, 2 and 1 respectively. FIDO-CA frugally assigns saliency to contextually important pixels while preserving classifier confidence (Figure 4), so we do not necessarily expect our saliency maps to correlate with the typically large human-labeled bounding boxes. The reliance on human-labeled bounding boxes makes WSL suboptimal for evaluating saliency maps, so we evaluate the so-called Saliency Metric proposed by BID2, which eschews the human labeled bounding boxes. The smallest bounding box A is computed as before. The image is then cropped using this bounding box and upscaling to its original size. The Saliency Metric is log max(Area(A), 0.05) − log p(c|CropAndUpscale(x, A)), the log ratio between the bounding box area and the in-class classifier probability after upscaling. This metric represents the information concentration about the label within the bounded region. From the superior performance of FIDO-CA we conclude that a strong generative model regularizes explanations towards the natural image manifold and finds concentrated region of features relevant to the classifier's prediction. FIG0: Examples from the ablation study. We show how each of our two innovations, FIDO and generative infilling, improve from previous methods that adopts BBMP with hueristics infilling (e.g. Blur and Random). Specifically, we compare with a new variant BBMP-CA that uses strong generative in-filling CA-GAN via thresholding the continous masks: we test a variety of decreasing thresholds. We find both FIDO (searching over Bernoulli masks) and generative in-filling (CAGAN) are needed to produce compact saliency maps (the right-most column) that retain class information. See Appendix B for more qualitative examples and in section A.7 for quantitative . Can existing algorithms be improved by adding an in-filling generative model without modeling a discrete distribution over per-feature masks? And does filling in the dropped-out region suffice without an expressive generative model? We carried out a ablation study that suggests no on both counts. We compare FIDO-CA to a BBMP variant that uses CA-GAN infilling (called BBMP-CA); we also evaluate FIDO with heuristic infilling (FIDO-Blur, FIDO-Random). Because the continuous mask of BBMP does not naturally partition the features into observed/unobserved, BBMP-CA first thresholds the masked region r = I(z > τ) before generating the reference φ(x r, x \r) with a sample from CA-GAN. We sweep the value of τ as 1, 0.7, 0.5, 0.3, 0.1 and 0. We find BBMP-CA is brittle with respect to its threshold value, producing either too spread-out or stringent saliency maps (FIG0). We observed that FIDO-Blur and FIDO-Random produce more concentrated saliency map than their BBMP counterpart with less artifacts, while FIDO-CA produces the most concentrated region on the target with fewest artifacts. Each of these baselines were evaluated on the two quantitative metrics (Appendix A.7); BBMP-CA considerably underperformed relative to FIDO-CA. Because the classifier behavior is ill-defined for out-of-distribution inputs, any explanation that relies on out-of-distribution feature values is unsatisfactory. By modeling the input distribution via an expressive generative model, we can encourage explanations that rely on counterfactual inputs close to the natural manifold. However, our performance is then upper-bounded by the ability of the generative model to capture the conditional input density. Fortunately, this bound will improve alongside future improvements in generative modeling. We proposed FIDO, a new framework for explaining differentiable classifiers that uses adaptive Bernoulli dropout with strong generative in-filling to combine the best properties of recently proposed methods BID3 BID2 BID18. We compute saliency by marginalizing over plausible alternative inputs, revealing concentrated pixel areas that preserve label information. By quantitative comparisons we find the FIDO saliency map provides more parsimonious explanations than existing methods. FIDO provides novel but relevant explanations for the classifier in question by highlighting contextual information relevant to the prediction and consistent with the training distribution. We released the code in PyTorch at https://github. com/zzzace2000/FIDO-saliency. Here we compare FIDO with two previously proposed methods, BBMP with Blur in-filling strategy BID3 and BBMP with Random in-filling strategy BID2. One potential concern in qualitatively comparing these methods is that each method might have a different sensitivity to the sparsity parameter λ. Subjectively, we observe that BBMP requires roughly 5 times higher sparsity penalty λ to get visually comparable saliency maps. In our comparisons we sweep λ over a reasonable range for each method and show the ing sequence of increasingly sparse saliency maps FIG0 ). We use λ = 5e−4, 1e−3, 2e−3, 5e−3.We observe that all methods are prone to artifacts in the low λ regime, so the appropriate selection of this value is clearly important. Interestingly, BBMP Blur and Random respectively find artifacts with different quality: small patches and pixels for Blur and structure off-object lines for Random. FIDO with CA is arguably the best saliency map, producing fewer artifacts and concentrating saliency on small regions for the images. Here we examine the effect of learning a reduced dimensionality θ that upsampled to the full image size during optimization. We consider a variety of upsampling rates, and in a slight abuse of terminology we refer to the upsampling "size" as the square root of the dimensionality of θ before upsampling, so smaller size implies more upsampling. In FIG0, we demonstrate two examples with different upsampling size under Mean and CA infilling methods with SSR objectives. The weaker infilling strategy Mean apparently requires stronger regularization to avoid artifacts compared to CA. Note that although CA produces much less artifacts compared to Mean, it still produces some small artifacts outside of the objects which is unfavored. We then choose 56 for the rest of our experiments to balance between details and the removal of the artifacts. FIG0: Comparisons of upsampling effect in Mean and CA infilling methods with no total variation penalty. We show the upsampling regularization removes the artifacts especially in the weaker infilling method Mean. To show the stability of our method, we test our method with different random seeds and observe if they are similar. In FIG0, our method produces similar saliency map for 4 different random seeds. Here we test the effect of total variation prior regularization in FIG0. We find the total variation can reduce the adversarial artifacts further, while risking losing signals when the total variation penalty is too strong. Here we quantitatively compare the in-filling strategies. The generative approaches (VAE and CA) perform visually sharper images than four other baselines. Since we expect this random removal should not remove the target information, we use the classification probability of the ResNet as our metric to measure how good the infilling method recover the target prediction. We quantitatively evaluate the probability for 1, 000 validation images in FIG0. We find that VAE and CA consistently outperform other methods, having higher target probability. We also note that all the heuristic baselines (Mean, Blur, Random) perform much worse since the heuristic nature of these approaches, the images they generate are not likely under the distribution of natural images leading to the poor performance by the classifier. FIG0: Box plot of the classifier probability under different infilling with respect to random masked pixels using ResNet under 1, 000 images. We show that generative models (VAE and CA) performs much better in terms of classifier probability. A.6 BATCH SIZE EFFECTS FIG0 shows the effect of batch size on the saliency map. We found unsatisfactory for batch size less than 4, which we attribute this to the high variance in the ing gradient estimates. We show the performance of BBMP-CA with various thresholds τ on both WSL and SM on subset of 1, 000 images in TAB3. We also show more qaulitative examples in FIG14. We find BBMP-CA is relatively brittle across different thresholds of τ. Though with τ = 0.3, the BBMP-CA perform slightly better than BBMP and FIDO with heuristics infilling, it still performs substantially inferior to FIDO-CA. We also perform the flipping experiment in FIG0 and show our FIDO-CA substantially outperforms BBMP-CA with varying different thresholds. B MORE EXAMPLES FIG0 shows several more infilled counterfactual images, along with the counterfactuals produced by the method from BID2. More examples comparing the various FIDO infilling approaches can be found in FIG12 and 21. BID2; FIDO-CA is our method with CA-GAN infilling BID16. Classifier confidence p(c|x) is reported below the input and each infilled image. We hypothesize by that FIDO-CA is able to isolate compact pixel areas of contextual information. For example, in the upper right image pixels in the net region around the fish are highlighted; this context information is missing from the Realtime saliency map but are apparently relevant to the classifier's prediction. These 4 examples are bulbul, tench, junco, and ptarmigan respectively.

represent the short summary for finding the corresponding scientific article (long).

We compute saliency by using a strong generative model to efficiently marginalize over plausible alternative inputs, revealing concentrated pixel areas that preserve label information.

represent the tldr for finding the corresponding scientific article

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This paper presents the Variation Network (VarNet), a generative model providing means to manipulate the high-level attributes of a given input. The originality of our approach is that VarNet is not only capable of handling pre-defined attributes but can also learn the relevant attributes of the dataset by itself. These two settings can be easily combined which makes VarNet applicable for a wide variety of tasks. Further, VarNet has a sound probabilistic interpretation which grants us with a novel way to navigate in the latent spaces as well as means to control how the attributes are learned. We demonstrate experimentally that this model is capable of performing interesting input manipulation and that the learned attributes are relevant and interpretable. We focus on the problem of generating variations of a given input in an intended way. This means that given some input element x, which can be considered as a template, we want to generate transformed versions of x with different high-level attributes. Such a mechanism is of great use in many domains such as image edition since it allows to edit images on a more abstract level and is of crucial importance for creative uses since it allows to generate new content. More precisely, given a dataset D = {(x, m ),..., (x (N), m (N) )} of N labeled elements (x, m) ∈ X × M, where X stands for the input space and M for the metadata space, we would like to obtain a model capable of learning a relevant attribute space Ψ ⊂ R d for some integer d > 0 and meaningful attribute functions φ: X × M → Ψ that we can then use to control generation. In a great majority of the recent proposed methods BID13; BID16, these attributes are assumed to be given. We identify two shortcomings: labeled data is not always available and this approach de facto excludes attributes that can be hard to formulate in an absolute way. The novelty of our approach is that these attributes can be either learned by the model (we name them free attributes) or imposed (fixed attributes). This problem is an ill-posed one on many aspects. Firstly, in the case of fixed attribute functions φ, there is no ground truth for variations since there is no x with two different attributes. Secondly, it can be hard to determine if a learned free attribute is relevant. However, we provide empirical evidence that our general approach is capable of learning such relevant attributes and that they can be used for generating meaningful variations. In this paper, we introduce the Variation Network (VarNet), a probabilistic neural network which provides means to manipulate an input by changing its high-level attributes. Our model has a sound probabilistic interpretation which makes the variations obtained by changing the attributes statistically meaningful. As a consequence, this probabilistic framework provides us with a novel mechanism to "control" or "shape" the learned free attributes which then gives interpretable controls over the variations. This architecture is general and provides a wide range of choices for the design of the attribute function φ: we can combine both free and fixed attributes and the fixed attributes can be either continuous or discrete. Our contributions are the following:• A widely applicable encoder-decoder architecture which generalizes existing approaches BID11; BID14; BID13 The input x,x are in X, the input space and the metadata m is in M, the metadata space. The latent template code z * lies in Z *, the template space, while the latent variable z lies in Z the latent space. The variable u is sampled from a zero-mean unitvariance normal distribution. Finally, the features φ(x, m) are in Ψ, the attribute space. The Neural Autoregressive Flows (NAF) BID10 are represented using two arrows, one pointing to the center of the other one; this denotes the fact that the actual parameters of first neural network are obtained by feeding meta-parameters into a second neural network. The discriminator D acts on Z * × Ψ.• An easy-to-use framework: any encoder-decoder architecture can be easily transformed into a VarNet in order to provide it with controlled input manipulation capabilities,• A novel and statistically sound approach to navigate in the latent space,• Ways to control the behavior of the free learned attributes. The plan of this paper is the following: Sect. 2 presents the VarNet architecture together with its training algorithm. For better clarity, we introduce separately all the components featured in our model and postpone the discussion about their interplay and the motivation behind our modeling choices in Sect. 3 and Sect. 4 discusses about the related works. In particular, we show that VarNet provides an interesting solution to many constrained generation problems already considered in the literature. Finally, we illustrate in Appendix A the possibilities offered by our proposed model and show that its faculty to generate variations in an intended way is of particular interest. We now introduce our novel encoder-decoder architecture which we name Variation Network. Our architecture borrows principles from the traditional Variational AutoEncoder (VAE) architecture BID11 and from the Wasserstein AutoEncoder (WAE) architecture BID15; BID14. It uses an adversarially learned regularization BID5; BID13, introduces a separate latent space for templates BID0 and decomposes the attributes on an adaptive basis BID17. It can be seen as a VAE with a particular decoder network or as a WAE with a particular encoder network. Our architecture is shown in FIG0 and our training algorithm is presented in Alg. 1.We detail in the following sections the different parts involved in our model. In Sect. 2.1, we focus on the encoder-decoder part of VarNet and explain Eq., and. In Sect. 2.2, we introduce the adversarially-learned regularization whose aim is to disentangle attributes from templates (Eq. FORMULA3 and FORMULA7). Section 2.3 discusses the special parametrization that we adopted for the attribute space Ψ. Require: DISPLAYFORM0, reconstruction cost c, reproducing kernel k, batch size n 1: for Fixed number of iterations do 2:Sample x:= (x 1, . . ., x n) and m:= (m 1, . . ., m n) where DISPLAYFORM1 Compute z:= {z 1, . . ., z n} where DISPLAYFORM2 Samplex:= {x 1, . . .,x n} wherex i ∼ p(·|z i), Sample random features {ψ i} i=1.. n from feature space Ψ using ν (see Sect. 2.3) 7:Letz:= {z 1, . . .,z n} wherez i ∼ p(·) Discriminator training phase DISPLAYFORM0 10:Gradient ascent step on the discriminator parameters using ∇L Disc Encoder-decoder training phase 12: DISPLAYFORM0 where DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 13:Gradient ascent step on all parameters except the discriminator parameters (encoder and decoder parameters, feature function parameters, features vectors and NAF f) using ∇L EncDec 14: end for Similar to the VAE architectures, we suppose that our data x ∈ X depends on some latent variable z ∈ Z through some decoder p(x|z) parametrized by a neural network. We introduce a prior p(z) over this latent space so that the joint probability distribution is expressed as p(x, z) = p(x|z)p(z). Since the posterior distribution p(z|x) is usually intractable, an approximate posterior distribution q(z|x) parametrized by a neural network is usually introduced. The novelty of our approach is on how we write this encoder network. Firstly, we introduce an attribute space Ψ ⊂ R d, where d is the dimension of the attribute space, on which we condition the encoder which we now denote as q(·|x, ψ ∈ Ψ). More details about the attribute space Ψ are given in Sect. 2.3. For the moment, we can consider it to be a subspace of R d from which we can sample from. The objective in doing so is that decoding z ∼ q(·|x, ψ) using p(x|z) will in a samplex that is a variation of x but with features ψ. Secondly, in order to correctly reconstruct x, introduce an attribute function φ: X × M → Ψ computed from x and its metadata m with values in the attribute space Ψ. This attribute function is a deterministic neural network that will be learned during training and whose aim is to compute attributes of x. For an input (x, m) ∈ D, we want to decouple a template obtained from x from its attributes φ(x, m) computed from x and (possibly) from its metadata m. This is done by introducing another latent space Z * that we term template space together with a approximated posterior distribution q * (z * |x)parametrized by a neural network and a fixed prior p * (z *). The idea is then to compute z from z * by applying a transformation parametrized only by the feature space Ψ. In practice, this is done by using a Neural Autoregressive Flow (NAF) BID10 f ψ: Z * → Z parametrized by ψ ∈ Ψ. Neural autoregressive flows are universal density estimation models which are capable of sampling any random variable Y by applying a learned transformation over a base random variable X (Thm. 1 in BID10).Given a reconstruction loss c on X, we have the following mean reconstruction loss: DISPLAYFORM0 We regularize the latent spaces Z * and Z by adding the usual KL term appearing in the VAE Evidence Lower Bound (ELBO) on Z *: DISPLAYFORM1 and an MMD-based regularization on Z similar the one used in WAEs (see Alg. 2 in BID15): DISPLAYFORM2 where k: Z ×Z → R is an positive-definite reproducing kernel and H k the associated Reproducing Kernel Hilbert Space (RKHS) BID1.The equations FORMULA5, FORMULA6 and of Alg. 1 are estimators on a mini-batch of size n of equations FORMULA8, FORMULA9 and FORMULA10 respectively, being the unbiased U-statistic estimator of BID7. Our encoder q(z|x, ψ) thus depends exclusively on x and on the feature space Ψ. However, there is no reason, for a random attribute ψ ∈ Ψ = φ(x, m), that p(x|z) where z ∼ q(z|x, φ) generates variations of the original x with features φ. Indeed, all needed information for reconstructing x is potentially already contained in z. We propose to add an adversarially-learned cost on the latent variable z * to force the encoder q * to discard information about the attributes of x: Specifically, we train a discriminator neural network D: Z * ×Ψ → whose role is to evaluate the probability D(z *, ψ) that there exists a (x, m) ∈ D such that ψ = φ(x, m) and z * ∼ q * (·|x). In other words, the aim of the discriminator is to determine if the attributes ψ and the template code z * originate from the same (x, m) ∈ D or if the features ψ are randomly generated. We postpone the explanation on how we sample random features ψ ∈ Ψ in Sect. 2.3 and suppose for the moment that we have access to a distribution ν(ψ) over Ψ from which we can sample. The encoder-decoder architecture presented in Sect. 2.1 is trained to fool the discriminator: this means that for a given (x, m) ∈ D it tries to produce a template code z * ∼ q * (·|x) which contains no information about the features φ(x, m).In an optimal setting, i.e. when the discriminator is unable to match any z * ∈ Z * with a particular feature ψ ∈ Ψ, the space of template codes and the space of attributes are decorrelated. All the missing information needed to reconstruct x given z * ∼ q * (·|x) lies in the transformation f φ(x,m). Since these transformations between the template space Z * and the latent space Z only depend on the feature space Ψ, they tend to be applicable over all template codes z * and generalize well. During generation time, it is then possible to change the attributes of a sample without changing its template. The discriminator is trained to maximize DISPLAYFORM0 while the encoder-decoder architecture is trained to minimize DISPLAYFORM1 Estimators of Eq. FORMULA3 and FORMULA3 are given by Eq. FORMULA3 and FORMULA7 respectively. We adopt a particular parametrization of our attribute function φ: X × M so that we are able to sample fake attributes without the need to rely on an existing (x, m) ∈ D pair. In the following, we make a distinction between two different cases: the case of continuous free attributes and the case of fixed continuous or discrete attributes. In order to handle free attributes, which denote attributes that are not specified a priori but learned. For this, we introduce d Ψ attribute vectors v i of dimension d together with an attention module α: X × M → dΨ, where d Ψ is the intrinsic dimension of the attribute space Ψ. By denoting α i the coordinates of α, we then write our attribute function φ as DISPLAYFORM0 This approach is similar to the style tokens approach presented in BID17. The v i's are global and do not depend on a particular instance (x, m). By varying the values of the α i's between, we can then span a d Ψ -dimensional hypercube in R d which stands for our attribute space Ψ. It is worth noting that the v i's are also learned and thus constitute an adaptive basis of the attribute space. In order to define a probability distribution ν over Ψ (note that this subspace also varies during training), we are free to choose any distribution ν α over dΨ. We then sample random attributes from ν by In the continuous case, we write our attribute function DISPLAYFORM1 DISPLAYFORM2 while in the discrete case, we just consider DISPLAYFORM3 where e m is a d Ψ -dimensional embedding of the symbol m. It is important to note that even if the attributes are fixed, the v i's or the embeddings e m are learned during training. These two equations define a natural probability distribution ν over Ψ: DISPLAYFORM4 We now detail our objective and notably explain our particular choice concerning the regularizations on the latent spaces Z * and Z. In Sect. 3.1, we will see that these insights suggest an additional way to "control" the influence of the learned free attributes. In Sect. 3.2, we further discuss about the multiple possibilities that we have concerning the implementation of the attribute function. We list, in Sect. 3.3, the different sampling schemes of VarNet. Finally, Sect. 3.4 is dedicated to implementation details. We discuss our choice concerning the regularizations of the latent spaces and specifically why we chose a KL regularization on Z * and an MMD loss on Z.We found that using a MMD-based regularization on the template space Z * ed in approximated posterior distributions q * (·|x) with very small variances (almost deterministic mappings). One explanation of this behavior is that the MMD regularization tries to enforce that the aggregated posterior DISPLAYFORM0 ) matches the prior p *: it does not act on the individual conditional probability distributions q * (·|x). This degenerate behavior is a side-effect of our adversarial regularization since stochastic encoders have been successfully used in WAEs BID14. When using the the Kullback-Leibler regularization on Z *, this effect disappear which makes the KL regularization that we considered more suited for VarNet since it helps to keep our model out of a degenerate regime. For some applications, it can still be of interest to have a control over the variance of the conditional probability distributions q * (·|x). Similar to the approach of BID9; BID2, we propose to multiply the KL term by a scalar parameter β > 0. For β = 1, we retrieve the original formulation. For β ∈]0, 1[, decreasing the value of β from one to zero decreases the variance of the q * (·|x). We found no gain in considering values of β greater than 1. Examples where this tuning provides an interesting application are given in Sect. A.2.We now consider the regularization over Z. This regularization is in fact superfluous and could be removed. However, we noticed that adding this MMD regularization helped obtaining better reconstruction losses. In this section, we focus on the parametrization of the attribute function φ: X × Z → R d and propose some useful use cases. The formulation of Sect. 2.3 is in fact too restrictive and considered only one attribute function. It is in fact possible to mix different attributes functions by simply concatenating the ing vectors. By doing so, we can then combine free and fixed attributes in a natural way but also consider different attention modules α. We can indeed use neural networks with different properties similarly to what is done in BID4 but also consider different distributions over the attention vectors α i.It is important to note that the free attributes presented in Sect. 2.3.1 can only capture global attributes, which are attributes that are relevant for all elements of the dataset D. In the presence of discrete labels m, it can be interesting to consider label-dependent free attributes, which are attributes specific to a subset of the dataset. In this case, the attribute function φ can be written as DISPLAYFORM0 where e m,i designates the i th attribute vector of the label m. With all these possibilities at hand, it is possible to devise numerous applications in which the notions of template and attribute of an input x may have diverse interpretations. Our choice of using a discriminator over Ψ instead of, for instance, over the values of α themselves allow to encompass within the same framework discrete and continuous fixed attributes. This makes the combinations of such attributes functions natural. We quickly review the different sampling schemes of VarNet. We believe that this wide range of usages makes VarNet a promising model for a wide range of applications. We can for instance:• generate random samplesx from the estimated dataset distribution: DISPLAYFORM0 • samplex with given attributes ψ: DISPLAYFORM1 • generate a variations of an input x with attributes ψ: DISPLAYFORM2 • generate random variations of an input x: x ∼ p(·|z) with z = f ψ (z *) where z * ∼ q * (·|x) and ψ ∼ ν(·).Note that for sampling generate random samplesx, we do that by sampling z * ∼ p * (·) from the prior, ψ ∼ ν(·) from the distribution of the attributes and then decoding z = f ψ (z *) decoding it using the decoder p(·|z) instead of just decoding a z * ∼ p * (·) sampled from the prior. This is due to the fact that, as already mentioned, this MMD regularization is not an essential element of the VarNet architecture: its role is more about fixing the "scale" of the Z space rather than enforcing that the aggregated posterior distribution exactly matches the prior. In the case of continuous attributes of the form Eq. FORMULA3 or FORMULA3, VarNet also provides a new way to navigate in the latent space Z. Indeed, for a given template latent code z *, it is possible to move continuously in the latent space Z by simply changing continuously the values of the α i and then DISPLAYFORM3 The image by the above transformation in the Z space of the d Ψ dimensional hypercube d ψ constitutes the space of variations of the template z *. Since our feature space bears a measure ν, this space of variations has a probabilistic interpretation. To the best of our knowledge, we think that it is the first time that a meaningful probabilistic interpretation about the displacement in the latent space in terms of attributes is given: We'll see in Appendix A.3 that two similar variations applied on different templates can induce radically different displacements in the latent space Z. We hope that this new technique will be useful in many applications and help go beyond the traditional (but unjustified) linear or spherical interpolations BID18. Our architecture is general and any decoder and encoder networks can be used. We chose to use a NAF 1 for our encoder network. This choice has the advantage of using a more expressive posterior distribution compared to the often-used diagonal Gaussian posterior distributions. Our priors p * and p are zero-mean unit-variance Gaussian distributions. For the MMD regularization, we used the parameters used in BID15 (λ = 10 and k(x, y) = C/(C+ x−y 2 2) the inverse multiquadratics kernel with C = 2dim(Z)). For the scalar coefficient γ, we found that a value of 10 worked well on all our experiments. For the sampling of the α values in the free attributes case, we considered ν α to be a uniform distribution over d ψ. In the fixed attribute case, we simply obtain a random sample {ψ i} n i=1 by shuffling the already computed batches of {φ(x i, m i)} n i=1 (lines 4 and 6 in Alg.1). The Variation Network generalizes many existing models used for controlled input manipulation by providing a unified probabilistic framework for this task. We now review the related literature and discuss the connections with VarNet. The problem of controlled input manipulation has been considered in the Fader networks paper BID13, where the authors are able to modify in a continuous manner the attributes of an input image. Similar to us, this approach uses an encoder-decoder architecture together with an adversarial loss used to decouple templates and attributes. The major difference with VarNet is that this model has a deterministic encoder which limits the sampling possibilities as discussed in Sect. A.2. Also, this approach can only deal with fixed attributes while VarNet is able to also learn meaningful free attributes. In fact, VAEs BID11,; BID14 and Fader networks can be seen as special cases of VarNet. Recently, the Style Tokens paper BID17 proposed a solution to learn relevant free attributes in the context of text-to-speech. The similarities with our approach is that the authors condition an encoder model on an adaptive basis of style tokens (what we called attribute space in this work). VarNet borrows this idea but cast it in a probabilistic framework, where a distribution over the attribute space is imposed and where the encoder is stochastic. Our approach also allows to take into account fixed attributes, which we saw can help shaping the free attributes. Traditional ways to explore the latent space of VAEs is by doing linear (or spherical BID18) interpolations between two points. However, there are two major caveats in this approach: the requirement of always needing two points in order to explore the latent space is cumbersome and the interpolation scheme is arbitrary and bears no probabilistic interpretation. Concerning the first point, a common approach is to find, a posteriori, directions in the latent space that accounts for a particular change of the (fixed) attributes BID16. These directions are then used to move in the latent space. Similarly, BID8 proposes a model where these directions of interest are given a priori. Concerning the second point, BID12 proposes to compute interpolation paths minimizing some energy functional which in interpolation curves rather than interpolation straight lines. However, this interpolation scheme is computationally demanding since an optimization problem must be solved for each point of the interpolation path. Another trend in controlled input manipulation is to make a posteriori analysis on a trained generative model BID6; BID0; BID16 BID3 using different means. One possible advantage of these methods compared to ours is that different attribute manipulations can be devised after the training of the generative model. But, these procedures are still costly and so provide any real-time applications where a user could provide on-the-fly the attributes they would like to modify. One of these approaches BID3 consists in using the trained decoder to obtained a mapping Z → X and then performing gradient descent on an objective which accounts for the constraints or change of the attributes. Another related approach proposed in BID6 consists in training a Generative Adversarial Network which learns to move in the vicinity of a given point in the latent space so that the decoded output enforces some constraints. The major difference of these two approaches with our work is that these movements are done in a unique latent space, while in our case we consider separate latent spaces. But more importantly, these approaches implicitly consider that the variation of interest lies in a neighborhood of the provided input. In BID0 the authors introduce an additional latent space called interpretable lens used to interpret the latent space of a generative model. This space shares similarity with our latent space Z * and they also propose a joint optimization for their model, where the encoder-decoder architecture and the interpretable lens are learned jointly. The difference with our approach is that the authors optimize an "interpretability" loss which requires labels and still need to perform a posteriori analysis to find relevant directions in the latent space. We presented the Variation Network, a generative model able to vary attributes of a given input. The novelty is that these attributes can be fixed or learned and have a sound probabilistic interpretation. Many sampling schemes have been presented together with a detailed discussion and examples. We hope that the flexibility in the design of the attribute function and the simplicity, from an implementation point of view, in transforming existing encoder-decoder architectures (it suffices to provide the encoder and decoder networks) will be of interest in many applications. For future work, we would like to extend our approach in two different ways: being able to deal with partially-given fixed attributes and handling discrete free attributes. We also want to investigate the of use stochastic attribute functions φ. Indeed, it appeared to us that using deterministic attribute functions was crucial and we would like to go deeper in the understanding of the interplay between all VarNet components. We now apply VarNet on MNIST in order to illustrate the different sampling schemes presented in Sect. A.In all these experiments, we choose to use a simple MLP with one hidden layer of size 400 for the encoder and decoder networks. We present and comment for different attribute functions and different sampling schemes. The different attribute functions we considered are• 1Free: one-dimensional free attribute space (Eq. FORMULA3 We display in Figure 2 samples obtained with the sampling procedures Eq. FORMULA3 From Fig. 2b, we see that the fixed label attribute have clearly been taken into account, but it can be hard to grasp which high-level attribute the free attribute function has captured. In order to visualize this, we plot in Fig. 3 a visualization of the space of variations spanned by a given template latent code z * . From these plots, it appears that the attribute vector encodes a notion of rotation meaningful for this digit dataset and it is interesting to note how different templates produce different "writing styles". Free attributes can thus be particularly interesting for capturing high-level features, such like rotation, that cannot be described in an absolute way or which are ill-defined. By observing carefully Fig. 3, we note that the variations generated by varying the free attribute applies to all digit classes, irrespective of their label. In such a case, it is impossible to obtain different "writing conventions" for the same digit (like cursive/printscript style for the digit "2") by only modifying the attributes. We show in FIG3 that, by considering free label-dependent attributes, we are able to smoothly go from one "writing convention" to the other one. We can gain further insight about the notion of template and attribute using the sampling scheme of Eq.. This sampling exploits the stochasticity of the encoder q * (·|x) in order to generate variations of a given input x using a fixed attribute ψ. An example of such variations is given in FIG4. The underlying idea is that, even for a given attribute ψ, there are multiple ways to generate variations of x with attributes ψ. We believe that this stochasticity is essential since, in many applications, there should not exist only one way to make variations. The parametrization of the attribute function has a crucial effect on the high-level features that they will able to capture. For instance, if we do not provide any label information, the information present in the template and the information contained in the attribute function can differ drastically. FIG5 show different space of variations where no label information is provided. The concepts captured in these cases are then related to thinness/roundness. Our intuition is that the free attributes capture the most general attributes of the dataset. For some applications, variation spaces such as the one displayed in FIG5, 6b or 6d are not desirable because they may tend to move too "far away" from the original input. As discussed in Sect. 3.1, it is possible to reduce how "spread" the spaces of variation are by modifying the β parameter multiplying the KL term in the objective Eq.. An example of such a variation space is displayed in FIG5.From all examples above, we see that our architecture is indeed capable of decoupling templates from learned attributes and that we have two ways of controlling the free attributes that are learned: by modifying the KL term in the objective Eq. and by carefully devising the attribute function. Indeed, the learned free attributes can capture different high-level features depending on the other fixed attributes they are coupled with. FIG5 and 6c display the space of variations using the 2Free attribute function for two different input. FIG5 display the space of variations using the 1Free attribute function. FIG5 was generated using a model trained with a low KL penalty (β = 0.1) VarNet proposes a novel solution to explore the latent spaces. Usual techniques to navigate in the space of VAEs such as interpolations or the use of attribute vectors (distinct from what we called attribute vectors in this work) are mostly intrinsically-based on moving using straight lines. This assumes that the underlying geometry is euclidean, which is not the case, and forgets about the probabilistic framework. Also, computing attribute vectors requires data with binary labels which are not always available. On the contrary, our approach grants a sound probabilistic interpretation of the attributes and the variations they generate. Indeed, when the discriminator is fooled by the encoder-decoder architecture, the attributes are distributed according to ν which has a simple interpretation (it is the push-forward of the ν α distribution which is considered to be a uniform distribution in all these examples). Also, thinking about variations as a subspace of smaller dimension than the whole latent space makes much sense for us. Figure 7 shows a visualization in the latent space Z of the variation spaces spanned by moving with constant steps in the attribute space Ψ. Two key elements appear: constant steps in the attribute space do not induce constant steps in the Z space and variation spaces are extremely diverse (they are not translated versions of a unique variation space). For us, this advocates for the fact that displacements in the latent spaces using straight lines have a priori no meaningful interpretation: the same change of attributes for two different inputs can lead to radically different displacements in the latent space. More generally, our proposition of parametrizing attribute-related displacements in a latent space using flows conditioned on a simpler space is appealing from a conceptual point of view since we do not mix, in the same latent space, its probabilistic interpretation given by the prior and its ability to grant meaningful ways to vary attributes.

Represent the short summary for finding the corresponding scientific article (long)

The Variation Network is a generative model able to learn high-level attributes without supervision that can then be used for controlled input manipulation.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Despite alarm over the reliance of machine learning systems on so-called spurious patterns in training data, the term lacks coherent meaning in standard statistical frameworks. However, the language of causality offers clarity: spurious associations are those due to a common cause (confounding) vs direct or indirect effects. In this paper, we focus on NLP, introducing methods and resources for training models insensitive to spurious patterns. Given documents and their initial labels, we task humans with revise each document to accord with a counterfactual target label, asking that the revised documents be internally coherent while avoiding any gratuitous changes. Interestingly, on sentiment analysis and natural language inference tasks, classifiers trained on original data fail on their counterfactually-revised counterparts and vice versa. Classifiers trained on combined datasets perform remarkably well, just shy of those specialized to either domain. While classifiers trained on either original or manipulated data alone are sensitive to spurious features (e.g., mentions of genre), models trained on the combined data are insensitive to this signal. We will publicly release both datasets. What makes a document's sentiment positive? What makes a loan applicant creditworthy? What makes a job candidate qualified? What about a photograph truly makes it depict a dolphin? Moreover, what does it mean for a feature to be relevant to such a determination? Statistical learning offers one framework for approaching these questions. First, we swap out the semantic question for a more readily answerable associative question. For example, instead of asking what comprises a document's sentiment, we recast the question as which documents are likely to be labeled as positive (or negative)? Then, in this associative framing, we interpret as relevant, those features that are most predictive of the label. However, despite the rapid adoption and undeniable commercial success of associative learning, this framing seems unsatisfying. Alongside deep learning's predictive wins, critical questions have piled up concerning spuriousness, artifacts, reliability, and discrimination, that the purely associative perspective appears ill-equipped to answer. For example, in computer vision, researchers have found that deep neural networks rely on surface-level texture or clues in the image's to recognize foreground objects even when that seems both unnecessary and somehow wrong: the beach is not what makes a seagull a seagull. And yet researchers struggle to articulate precisely why models should not rely on such patterns. In NLP, these issues have emerged as central concerns in the literature on annotation artifacts and bias (in the societal sense). Across myriad tasks, researchers have demonstrated that models tend to rely on spurious associations (; ; ;). Notably, some models for question-answering tasks may not actually be sensitive to the choice of the question , while in Natural Language Inference (NLI), classifiers trained on hypotheses only (vs hypotheses and premises) perform surprisingly well . However, papers seldom make clear what, if anything, spuriousness means within the standard supervised learning framework. ML systems are trained to exploit the mutual information between features and a label to make accurate predictions. Statistical learning does not offer a conceptual distinction between between spurious and non-spurious associations. Causality, however, offers a coherent notion of spuriousness. Spurious associations owe to common cause rather than to a (direct or indirect) causal path. We might consider a factor of variation to be spuriously correlated with a label of interest if intervening upon it (counterfactually) would not impact the applicability of the label or vice versa. While our paper does not rely on the mathematical machinery of causality, we draw inspiration from the underlying philosophy to design a new dataset creation procedure in which humans counterfactually augment datasets. Returning to NLP, even though the raw data does not come neatly disentangled into manipulable factors, people nevertheless speak colloquially of editing documents to manipulate specific aspects . For example, the following interventions seem natural: (i) Revise the letter to make it more positive; (ii) Edit the second sentence so that it appears to contradict the first. The very notion of targeted revisions like (i) suggests a generative process in which the sentiment is but one (manipulable) cause of the final document. These edits might be thought of as intervening on sentiment while holding all upstream features constant. However even if some other factor has no influence on sentiment, if they share some underlying common cause (confounding), then we might expect aspects of the final document to be predictive of sentiment owing to spurious association. In this exploratory paper, we design a human-in-the-loop system for counterfactually manipulating documents. Our hope is that by intervening only upon the factor of interest, we might disentangle the spurious and non-spurious associations, yielding classifiers that hold up better when spurious associations do not transport out of sample. We employ crowd workers not to label documents, but rather to edit them, manipulating the text to make a targeted (counterfactual) class apply. For sentiment analysis, we direct the worker: revise this negative movie review to make it positive, without making any gratuitous changes. We might regard the second part of this directive as a sort of least action principle, ensuring that we perturb only those spans necessary to alter the applicability of the label. For NLI, a 3-class classification task (entailment, contradiction, neutral), we ask the workers to modify the premise while keeping the hypothesis intact, and vice versa, seeking two sets of edits corresponding to each of the (two) counterfactual classes. Using this platform, we collect thousands of counterfactually-manipulated examples for both sentiment analysis and NLI, extending the IMDb and SNLI datasets, respectively. The is two new datasets (each an extension of a standard resource) that enable us to both probe fundamental properties of language and train classifiers less reliant on spurious signal. We show that classifiers trained on original IMDb reviews fail on counterfactually-revised data and vice versa. We further show that spurious correlations in these datasets are picked up by even linear models, however, augmenting the revised examples breaks up these correlations (e.g., genre ceases to be predictive of sentiment). For a Bidirectional LSTM trained on IMDb reviews, classification accuracy goes down from 79.3% to 55.7% when evaluated on original vs revised reviews. The same classifier trained on revised reviews achieves an accuracy of 62.5% on original reviews compared to 89.1% on their revised counterparts. These numbers go to 81.7% and 92.0% respectively when the classifier is retrained on the combined dataset. Similar behavior is observed for linear classifiers. We discovered that BERT is more resilient to such drops in performance on sentiment analysis. Despite that, it appears to rely on spurious associations in SNLI hypotheses identified by. We show that if fine-tuned on SNLI sentence pairs, BERT fails on pairs with revised premise and vice versa, experiencing more than a 30 point drop in accuracy. However, fine-tuned on the combined set, it performs much better across all datasets. Similarly, a Bi-LSTM trained on hypotheses alone can accurately classify 69% of the SNLI dataset but performs worse than the majority class baseline when evaluated on the revised dataset. When trained on hypotheses only from the combined dataset, its performance is expectedly worse than simply selecting the majority class on both SNLI as well as the revised dataset. Several papers demonstrate cases where NLP systems appear not to learn what humans consider to be the difference that makes the difference. For example, otherwise state-of-the-art models have been shown to be vulnerable to synthetic transformations such as distractor phrases , to misclassify paraphrased task ) and to fail on template-based modifications . demonstrate that simply replacing words by synonyms or hypernyms, which should not alter the applicable label, nevertheless breaks ML-based NLI systems. and show that classifiers correctly classified the hypotheses alone in about 69% of SNLI corpus. They further discover that crowd workers adopted specific annotation strategies and heuristics for data generation. identify similar issues exist with automatically-constructed benchmarks for question-answering . discover that reported numbers in question-answering benchmarks could often be achieved by the same models when restricted to be blind either to the question or to the passages.; and showed how imbalances in training data lead to unintended bias in the ing models, and, consequently, potentially unfair applications. substitute words to test the behavior of sentiment analysis algorithms in the presence of stylistic variation, finding that similar word pairs produce significant differences in sentiment score. Several papers explore richer feedback mechanisms for classification. Some ask annotators to highlight rationales, spans of text indicative of the label (; ;). For each document, remove the rationales to generate contrast documents, learning classifiers to distinguish original documents from their contrasting counterparts. While this feedback is easier to collect than ours, how to leverage it for training deep NLP models, where features are not neatly separated, remains less clear. programmatically alter text to invert gender bias and combined the original and manipulated data yielding gender-balanced dataset for learning word embeddings. In the simplest experiments, they swap each gendered word for its other-gendered counterpart. For example, the doctor ran because he is late becomes the doctor ran because she is late. However, they do not substitute names even if they co-refer to a gendered pronoun. Building on their work, describe a data augmentation approach for mitigating gender stereotypes associated with animate nouns for morphologically-rich languages like Spanish and Hebrew. They use a Markov random field to infer how the sentence must be modified while altering the grammatical gender of particular nouns to preserve morpho-syntactic agreement. In contrast, describe a method for probabilistic automatic in-place substitution of gendered words in a corpus. Unlike Lu et al., they propose an explicit treatment of first names by pre-defining name-pairs for swapping, thus expanding Lu et al.'s list of gendered word pairs significantly. We use Amazon's Mechanical Turk crowdsourcing platform to recruit editors to counterfactually revise each dataset. To ensure high quality of the collected data, we restricted the pool to U.S. residents that had already completed at least 500 HITs and had an over 97% HIT approval rate. For each HIT, we conducted pilot tests to identify appropriate compensation per assignment, receive feedback from workers and revise our instructions accordingly. A total of 713 workers contributed throughout the whole process, of which 518 contributed edits reflected in the final datasets. The original IMDb dataset consists of 50000 reviews divided equally across train and test splits. To keep the task of editing from growing unwieldy, we filter out the longest 20% of reviews, leaving 20000 reviews in the train split from which we randomly sample 2500 reviews, enforcing a 50:50 class balance. Following revision by the crowd workers, we partition this dataset into train/validation/test splits containing 1707, 245 and 488 examples, respectively. We present each review to two workers, instructing to revise the review such that (a) the document remains coherent and (b) the new label (given) accurately describes the revised document. Moreover, we instruct the workers not to make gratuitous modifications. Over a four week period, we manually inspected each generated review and rejected the ones that were outright wrong (sentiment was still the same or the review was a spam). After review, we rejected roughly 2% of revised reviews. For 60 original reviews, we did not approve any among the counterfactually-revised counterparts supplied by the workers. To construct the new dataset, we chose one revised review (at random) corresponding to each original review. In qualitative analysis, we identified eight common patterns among the edits (Table 2). For each review, having access to its counterfactually-revised counterpart enables us to isolate which parts the review humans believe are truly indicative of sentiment. These are the parts that were removed, replaced, or inserted into the original review to generate a new review that has the opposite sentiment. We identify the position indices where such replacements or insertions were made and create a binary vector representing the edits in each original review. To analyze inter-editor agreement, we compute the Jaccard similarity between the vectors corresponding to each revised review (Table 1). We observe that there is a higher agreement between two workers on smaller reviews and it decreases with the length of the review. Natural Language Inference Unlike sentiment analysis, SNLI is 3-way classification task, with inputs consisting of two sentences, a premise and a hypothesis and the three possible labels being entailment, contradiction, and neutral. The label is meant to describe the relationship between the facts stated in each sentence. We randomly sampled 1750, 250, and 500 pairs from the train, validation, and test sets of SNLI respectively, constraining the new data to have balanced classes. In Table 2: Most prominent categories of edits performed by humans for sentiment analysis (Original/Revised, in order). Red spans were replaced by Blue spans. Recasting fact as hoped for The world of Atlantis, hidden beneath the earth's core, is fantastic The world of Atlantis, hidden beneath the earth's core is supposed to be fantastic Suggesting sarcasm thoroughly captivating thriller-drama, taking a deep and realistic view thoroughly mind numbing "thriller-drama", taking a "deep" and "realistic" (who are they kidding?) view Inserting modifiers The presentation of simply Atlantis' landscape and setting The presentation of Atlantis' predictable landscape and setting Replacing modifiers "Election" is a highly fascinating and thoroughly captivating thriller-drama "Election" is a highly expected and thoroughly mind numbing "thriller-drama" Inserting phrases Although there's hardly any action, the ending is still shocking. Although there's hardly any action (or reason to continue watching past 10 minutes), the ending is still shocking. Diminishing via qualifiers which, while usually containing some reminder of harshness, become more and more intriguing. which, usually containing some reminder of harshness, became only slightly more intriguing. Granted, not all of the story makes full sense, but the film doesn't feature any amazing new computer-generated visual effects. Granted, some of the story makes sense, but the film doesn't feature any amazing new computer-generated visual effects. Changing ratings one of the worst ever scenes in a sports movie. 3 stars out of 10. one of the wildest ever scenes in a sports movie. 8 stars out of 10. one HIT, we asked workers to revise the hypothesis while keeping the premise intact, seeking edits corresponding to each of the two counterfactual classes. We refer to this data as Revised Hypothesis (RH). In another HIT, we asked workers to revise the original premise, while leaving the original hypothesis intact, seeking similar edits, calling it Revised Premise (RP). Following data collection, we employed a different set of workers to verify whether the given label accurately described the relationship between each premise-hypothesis pair. We presented each pair to three workers and performed a majority vote. When all three reviewers were in agreement, we approved or rejected the pair based on their decision, else, we verified the data ourselves. Finally, we only kept premise-hypothesis pairs for which we had valid revised data in both RP and RH, corresponding to both counterfactual labels. As a , we discarded ≈ 9% data. RP and RH, each comprised of 3332 pairs in train, 400 in validation, and 800 in test, leading to a total of 6664 pairs in train, 800 in validation, and 1600 in test in the revised dataset. In qualitative analysis, we identified some common patterns among hypothesis and premise edits (Table 3, 4). We collected all data after IRB approval and measured the time taken to complete each HIT to ensure that all workers were paid more than the federal minimum wage. During our pilot studies, workers spent roughly 5 minutes per revised review, and 4 minutes per revised sentence (for NLI). We paid workers $0.65 per revision, and $0.15 per verification, totalling $10778.14 for the study. Our experiments rely on the following five models: Support Vector Machines (SVMs), Naïve Bayes (NB) classifiers, Bidirectional Long Short-Term Memory Networks (Bi-LSTMs; Graves & Schmid- huber, 2005), ELMo models with LSTM, and fine-tuned BERT models . For brevity, we discuss only implementation details necessary for reproducibility. We use scikit-learn implementations of SVMs and Naïve Bayes for sentiment analysis. We train these models on TF-IDF bag of words feature representations of the reviews. We identify parameters for both classifiers using grid search conducted over the validation set. Bi-LSTM When training Bi-LSTMs for sentiment analysis, we restrict the vocabulary to the most frequent 20000 tokens, replacing out of vocabulary tokens by UNK. We fix the maximum input length at 300 tokens and pad smaller reviews. Each token is represented by a randomly-initialized 50-dimensional embedding. Our model consists of a bidirectional LSTM (hidden size 50) with recurrent dropout (probability 0.5) and global max-pooling following the embedding layer. To generate output, we feed this (fixed-length) representation through a fully-connected hidden layer with ReLU activation (hidden size 50), and then a fully-connected output layer with softmax activation. We train all models for a maximum of 20 epochs using Adam , with a learning rate of 1e-3 and a batch size of 32. We apply early stopping when validation loss does not decrease for 5 epochs. We also experimented with a larger Bi-LSTM which led to overfitting. We use the architecture described in to evaluate hypothesis-only baselines. 1 ELMo-LSTM We compute contextualized word representations (ELMo) using character-based word representations and bidirectional LSTMs . The module outputs a 1024- dimensional weighted sum of representations from the 3 Bi-LSTM layers used in ELMo. We represent each word by a 128-dimensional embedding concatenated to the ing 1024-dimensional ELMo representation, leading to a 1152-dimensional hidden representation. Following Batch Normalization, this is passed through an LSTM (hidden size 128) with recurrent dropout (probability 0.2). The output from this LSTM is then passed to a fully-connected output layer with softmax activation. We train this model for up to 20 epochs with same early stopping criteria as for Bi-LSTM, using the Adam optimizer with a learning rate of 1e-3 and a batch size of 32. BERT We use an off-the-shelf uncased BERT Base model, fine-tuning for each task. 2 To account for BERT's sub-word tokenization, we set the maximum token length is set at 350 for sentiment analysis and 50 for NLI. We fine-tune BERT up to 20 epochs with same early stopping criteria as for Bi-LSTM, using the BERT Adam optimizer with a batch size of 16 (to fit on a Tesla V-100 GPU). We found learning rates of 5e-5 and 1e-5 to work best for sentiment analysis and NLI respectively. We find that for sentiment analysis, linear models trained on the original 1.7k reviews achieve 80% accuracy when evaluated on original reviews but only 51% (level of random guessing) on revised reviews (Table 5). Linear models trained on revised reviews achieve 91% accuracy on revised reviews but only 58.3% on the original test set. We see similar pattern for Bi-LSTMs where accuracy drops substantially in both directions. Interestingly, while BERT models suffer drops too, they are less pronounced, perhaps a benefit of the exposure to a larger dataset where the spurious patterns may not have held. Classifiers trained on combined datasets perform well on both, often within ≈ 3 pts of models trained on the same amount of data taken only from the original distribution. Thus, there may be a price to pay for breaking the reliance on spurious associations, but it may not be substantial. We also conduct experiments to evaluate our sentiment models vis-a-vis their generalization out-ofsample to new domains. We evaluate models on Amazon reviews on data aggregated over six genres: beauty, fashion, appliances, giftcards, magazines, and software, the Twitter sentiment dataset , 3 and Yelp reviews released as part of the Yelp dataset challenge. We show that in almost all cases, models trained on the counterfactually-augmented IMDb dataset perform better than models trained on comparable quantities of original data. To gain intuition about what is learnable absent the edited spans, we tried training several models on passages where the edited spans have been removed from training set sentences (but not test set). SVM, Naïve Bayes, and Bi-LSTM achieve 57.8%, 59.1%, 60.2% accuracy, respectively, on this task, suggesting that there is substantial signal in these potentially immaterial sections. However, BERT performs worse than random guessing. In one simple demonstration of the benefits of our approach, we note that seemingly irrelevant words such as: romantic, will, my, has, especially, life, works, both, it, its, lives and gives (correlated with positive sentiment), and horror, own, jesus, cannot, even, instead, minutes, your, effort, script, seems and something (correlated with negative sentiment) are picked up as high-weight features by linear models trained on either original or revised reviews as top predictors. However, because humans never edit these during revision owing to their lack of semantic relevance, combining the original and revised datasets breaks these associations and these terms cease to be predictive of sentiment (Fig 4). Models trained on original data but at the same scale as combined data are able to perform slightly better on the original test set but still fail on the revised reviews. All models trained on 19k original reviews receive a slight boost in accuracy on revised data (except Naïve Bayes), yet their performance significantly worse compared to specialized models. Retraining models on a combination of the original 19k reviews with revised 1.7k reviews leads to significant increases in accuracy for all models on classifying revised reviews, while slightly improving the accuracy on classifying the original reviews. This underscores the importance of including counterfactuallyrevised examples in training data. Natural Language Inference Fine-tuned on 1.67k original sentence pairs, BERT achieves 72.2% accuracy on SNLI dataset but it is only able to accurately classify 39.7% sentence pairs from the RP set (Table 7). Fine-tuning BERT on the full SNLI training set (500k sentence pairs) in similar behavior. Fine-tuning it on RP sentence pairs improves its accuracy to 66.3% on RP but causes a drop of roughly 20 pts on SNLI. On RH sentence pairs, this in an accuracy of 67% on RH and 71.9% on SNLI test set but 47.4% on the RP set. To put these numbers in context, each individual hypothesis sentence in RP is associated with two labels, each in the presence of a different premise. A model that relies on hypotheses only would at best perform slightly better than choosing the majority class when evaluated on this dataset. However, fine-tuning BERT on a combination of RP and RH leads to consistent performance on all datasets as the dataset design forces models to look at both premise and hypothesis. Combining original sentences with RP and RH improves these numbers even further. We compare this with the performance obtained by fine-tuning it on 8.3k sentence pairs sampled from SNLI training set, and show that while the two perform roughly within 4 pts of each other when evaluated on SNLI, the former outperforms latter on both RP and RH. To further isolate this effect, Bi-LSTM trained on SNLI hypotheses only achieves 69% accuracy on SNLI test set, which drops to 44% if it is retrained on combination of original, RP and RH data (Table 8). Note that this combined dataset consists of five variants of each original premisehypothesis pair. Of these five pairs, three consist of the same hypothesis sentence, each associated with different truth value given the respective premise. Using these hypotheses only would provide conflicting feedback to a classifier during training, thus causing the drop in performance. Further, we notice that the gain of the latter over majority class baseline comes primarily from the original data, as the same model retrained only on RP and RH data experiences a further drop of 11.6% in accuracy, performing worse than just choosing the majority class at all times. One reasonable concern might be that our models would simply distinguish whether an example were from the original or revised dataset and thereafter treat them differently. The fear might be that our models would exhibit a hypersensitivity (rather than insensitivity) to domain. To test the potential for this behavior, we train several models to distinguish between original and revised data (Table 9). BERT identifies original reviews from revised reviews with 77.3% accuracy. In case of NLI, BERT and Naïve Bayes perform roughly within 3 pts of the majority class baseline (66.7%) whereas SVM performs substantially worse. By leveraging humans not only to provide labels but also to intervene upon the data, revising documents to alter the applicability of various labels, we are able to derive insights about the underlying semantic concepts. Moreover we can leverage the augmented data to train classifiers less dependent on spurious associations. Our study demonstrates the promise of leveraging human-in-the-loop feedback to disentangle the spurious and non-spurious associations, yielding classifiers that hold up better when spurious associations do not transport out of sample. Our methods appear useful on both sentiment analysis and NLI, two contrasting tasks. In sentiment analysis, expressions of opinion matter more than stated facts, while in NLI this is reversed. SNLI poses another challenge in that it is a 3-class classification task using two input sentences. In future work, we plan to extend these techniques, finding ways to leverage humans in the loop to build more robust systems for question answering and summarization (among others).

represent the short summary for finding the corresponding scientific article (long)

Humans in the loop revise documents to accord with counterfactual labels, resulting resource helps to reduce reliance on spurious associations.

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.