Question: Represent the article for finding a single-sentence TLDR

Question: We study the problem of learning and optimizing through physical simulations via differentiable programming. We present DiffSim, a new differentiable programming language tailored for building high-performance differentiable physical simulations. We demonstrate the performance and productivity of our language in gradient-based learning and optimization tasks on 10 different physical simulators. For example, a differentiable elastic object simulator written in our language is 4.6x faster than the hand-engineered CUDA version yet runs as fast, and is 188x faster than TensorFlow. Using our differentiable programs, neural network controllers are typically optimized within only tens of iterations. Finally, we share the lessons learned from our experience developing these simulators, that is, differentiating physical simulators does not always yield useful gradients of the physical system being simulated. We systematically study the underlying reasons and propose solutions to improve gradient quality. Figure 1: Left: Our language allows us to seamlessly integrate a neural network (NN) controller and a physical simulation module, and update the weights of the controller or the initial state parameterization (blue). Our simulations typically have 512 ∼ 2048 time steps, and each time step has up to one thousand parallel operations. Right: 10 differentiable simulators built with DiffSim. Differentiable physical simulators are effective components in machine learning systems. For example, de Avila Belbute- Peres et al. (2018a) and Hu et al. (2019b) have shown that controller optimization with differentiable simulators converges one to four orders of magnitude faster than model-free reinforcement learning algorithms. The presence of differentiable physical simulators in the inner loop of these applications makes their performance vitally important. Unfortunately, using existing tools it is difficult to implement these simulators with high performance. We present DiffSim, a new differentiable programming language for high performance physical simulations on both CPU and GPU. It is based on the Taichi programming language (a). The DiffSim automatic differentiation system is designed to suit key language features required by physical simulation, yet often missing in existing differentiable programming tools, as detailed below: Megakernels Our language uses a "megakernel" approach, allowing the programmer to naturally fuse multiple stages of computation into a single kernel, which is later differentiated using source code transformations and just-in-time compilation. Compared to the linear algebra operators in TensorFlow and PyTorch , DiffSim kernels have higher arithmetic intensity and are therefore more efficient for physical simulation tasks. Imperative Parallel Programming In contrast to functional array programming languages that are popular in modern deep learning (; ; b), most traditional physical simulation programs are written in imperative languages such as Fortran and C++. DiffSim likewise adopts an imperative approach. The language provides parallel loops and control flows (such as "if" statements), which are widely used constructs in physical simulations: they simplify common tasks such as handling collisions, evaluating boundary conditions, and building iterative solvers. Using an imperative style makes it easier to port existing physical simulation code to DiffSim. Flexible Indexing Existing parallel differentiable programming systems provide element-wise operations on arrays of the same shape, e.g. can only be expressed with unintuitive scatter/gather operations in these existing systems, which are not only inefficient but also hard to develop and maintain. On the other hand, in DiffSim, the programmer directly manipulates array elements via arbitrary indexing, thus allowing partial updates of global arrays and making these common simulation patterns naturally expressible. The explicit indexing syntax also makes it easy for the compiler to perform access optimizations (a). The three requirements motivated us to design a tailored two-scale automatic differentiation system, which makes DiffSim especially suitable for developing complex and high-performance differentiable physical simulators, possibly with neural network controllers (Fig. 1, left). Using our language, we are able to quickly implement and automatically differentiate 10 physical simulators 1, covering rigid bodies, deformable objects, and fluids (Fig. 1, right). A comprehensive comparison between DiffSim and other differentiable programming tools is in Appendix A. DiffSim is based on the Taichi programming language (a). Taichi is an imperative programming language embedded in C++14. It delivers both high performance and high productivity on modern hardware. The key design that distinguishes Taichi from other imperative programming languages such as C++/CUDA is the decoupling of computation from data structures. This allows programmers to easily switch between different data layouts and access data structures with indices (i.e. x[i, j, k]), as if they are normal dense arrays, regardless of the underlying layout. The Taichi compiler then takes both the data structure and algorithm information to apply performance optimizations. Taichi provides "parallel-for" loops as a first-class construct. These designs make Taichi especially suitable for writing high-performance physical simulators. For more details, readers are referred to Hu et al. (2019a). The DiffSim language frontend is embedded in Python, and a Python AST transformer compiles DiffSim code to Taichi intermediate representation (IR). Unlike Python, the DiffSim language is compiled, statically-typed, parallel, and differentiable. We extend the Taichi compiler to further compile and automatically differentiate the generated Taichi IR into forward and backward executables. We demonstrate the language using a mass-spring simulator, with three springs and three mass points, as shown right. In this section we introduce the forward simulator using the DiffSim frontend of Taichi, which is an easier-to-use wrapper of the Taichi C++14 frontend. Allocating Global Variables Firstly we allocate a set of global tensors to store the simulation state. These tensors include a scalar loss of type float32, 2D tensors x, v, force of size steps ×n_springs and type float32x2, and 1D arrays of size n_spring for spring properties: spring_anchor_a (int32), spring_anchor_b (int32), spring_length (float32). Defining Kernels A mass-spring system is modeled by Hooke's law where k is the spring stiffness, F is spring force, x a and x b are the positions of two mass points, and l 0 is the rest length. The following kernel loops over all the springs and scatters forces to mass points: For each particle i, we use semi-implicit Euler time integration with damping:, m i are the velocity, position and mass of particle i at time step t, respectively. α is a damping factor. The kernel is as follows: The main goal of DiffSim's automatic differentiation (AD) system is to generate gradient simulators automatically with minimal code changes to the traditional forward simulators. Design Decision Source Code Transformation (SCT) and Tracing are common choices when designing AD systems. In our setting, using SCT to differentiate a whole simulator with thousands of time steps, in high performance yet poor flexibility and long compilation time. On the other hand, naively adopting tracing provides flexibility yet poor performance, since the "megakernel" structure is not preserved during backpropagation. To get both performance and flexibility, we developed a two-scale automatic differentiation system (Figure 2): we use SCT for differentiating within kernels, and use a light-weight tape that only stores function pointers and arguments for end-to-end simulation differentiation. The global tensors are natural checkpoints for gradient evaluation. Assumption Unlike functional programming languages where immutable output buffers are generated, imperative programming allows programmers to freely modify global tensors. To make automatic differentiation well-defined under this setting, we make the following assumption on imperative kernels: Figure 2: Left: The DiffSim system. We reuse some infrastructure (white boxes) from Taichi, while the blue boxes are our extensions for differentiable programming. Right: The tape records kernel launches and replays the gradient kernels in reverse order during backpropagation. Global Data Access Rules: 1) If a global tensor element is written more than once, then starting from the second write, the write must come in the form of an atomic add ("accumulation"). 2) No read accesses happen to a global tensor element, until its accumulation is done. In forward simulators, programmers may make subtle changes to satisfy the rules. For instance, in the mass-spring simulation example, we record the whole history of x and v, instead of keeping only the latest values. The memory consumption issues caused by this can be alleviated via checkpointing, as discussed later in Appendix D. With these assumptions, kernels will not overwrite the outputs of each other, and the goal of AD is clear: given a primal kernel f that takes as input X 1, X 2,..., X n and outputs (or accumulates to) Users can specify the storage of adjoint tensors using the Taichi data structure description language (a), as if they are primal tensors. We also provide ti.root.lazy_grad to automatically place the adjoint tensors following the layout of their primals. A typical Taichi kernel consists of multiple levels of for loops and a body block. To make later AD easier, we introduce two basic code transforms to simplify the loop body, as detailed below. Flatten Branching In physical simulation branches are common, e.g. when implementing boundary conditions and collisions. To simplify the reverse-mode AD pass, we first flatten "if" statements by replacing every instruction that leads to side effects with the ternary operator select(cond, value_if_true, value_if_false) -whose gradient is clearly defined -and a store instruction (Fig. 3, middle). This is a common transformation in program vectorization (e.g. ;). Eliminate Mutable Local Variables After removing branching, we end up with straight-line loop bodies. To further simplify the IR and make the procedure truly single-assignment, we apply a series of local variable store forwarding transforms, until the mutable local variables can be fully eliminated (Fig. 3, right). After these two custom IR simplification transforms, DiffSim only has to differentiate the straightline code without mutable variables, which it achieves with reverse-mode AD, using a standard source code transformation . More details on this transform are in Appendix B. Loops Most loops in physical simulation are parallel loops, and during AD we preserve the parallel loop structures. For loops that are not explicitly marked as parallel, we reverse the loop order during AD transforms. We do not support loops that carry a mutating local variable since that would require a complex and costly run-time stack to maintain the history of local variables. Instead, users are instructed to employ global variables that satisfy the global data access rules. Parallelism and Thread Safety For forward simulation, we inherit the "parallel-for" construct from Taichi to map each loop iteration onto CPU/GPU threads. Programmers use atomic operations for thread safety. Our system can automatically differentiate these atomic operations. Gradient contributions in backward kernels are accumulated to the adjoint tensors via atomic adds. We construct a tape (Fig. 2, right) of the kernel execution so that gradient kernels can be replayed in a reversed order. The tape is very light-weight: since the intermediate are stored in global tensors, during forward simulation the tape only records kernel names and the (scalar) input parameters, unlike other differentiable functional array systems where all the intermediate buffers have to be recorded by the tape. Whenever a DiffSim kernel is launched, we append the kernel function pointer and parameters to the tape. When evaluating gradients, we traverse the reversed tape, and invoke the gradient kernels with the recorded parameters. Note that DiffSim AD is evaluating gradients with respect to input global tensors instead of the input parameters. Learning/Optimization with Gradients Now we revisit the mass-spring example and make it differentiable for optimization. Suppose the goal is to optimize the rest lengths of the springs so that the triangle area formed by the three springs becomes 0.2 at the end of the simulation. We first define the loss function: Taichi Complex Kernels Sometimes the user may want to override the gradients provided by the compiler. For example, when differentiating a 3D singular value decomposition done with an iterative solver, it is better to use a manually engineered SVD derivative subroutine for better stability. We provide two more decorators ti.complex_kernel and ti.complex_kernel_grad to overwrite the default automatic differentiation, as detailed in Appendix C. Apart from custom gradients, complex kernels can also be used to implement checkpointing, as detailed in Appendix D. We evaluate DiffSim on 10 different physical simulators covering large-scale continuum and smallscale rigid body simulations. All can be reproduced with the provided script. The dynamic/optimization processes are visualized in the supplemental video. In this section we focus our discussions on three simulators. More details on the simulators are in Appendix E. First, we build a differentiable continuum simulation for soft robotics applications. The physical system is governed by momentum and mass conservation, i.e. ρ We follow ChainQueen's implementation (b) and use the moving least squares material point method to simulate the system. We were able to easily translate the original CUDA simulator into DiffSim syntax. Using this simulator and an open-loop controller, we can easily train a soft robot to move forward (Fig. 1, diffmpm). Performance and Productivity Compared with manual gradient implementations in (b), getting gradients in DiffSim is effortless. As a , the DiffSim implementation is 4.2× shorter in terms of lines of code, and runs almost as fast; compared with TensorFlow, DiffSim code is 1.7× shorter and 188× faster (Table 1). The Tensorflow implementation is verbose due to the heavy use of tf.gather_nd/scatter_nd and array transposing and broadcasting. We implemented a smoke simulator (Fig. 1, smoke) with semi-Lagrangian advection and implicit pressure projection, following the example in Autograd . Using gradient descent optimization on the initial velocity field, we are able to find a velocity field that changes the pattern of the fluid to a target image (Fig. 7a in Appendix). We compare the performance of our system against PyTorch, Autograd, and JAX in Table 2. Note that as an example from the Autograd library, this grid-based simulator is intentionally simplified to suit traditional array-based programs. For example, a periodic boundary condition is used so that Autograd can represent it using numpy.roll, without any branching. Still, Taichi delivers higher performance than these arraybased systems. The whole program takes 10 seconds to run in DiffSim on a GPU, and 2 seconds are spent on JIT. JAX JIT compilation takes 2 minutes. We built an impulse-based differentiable rigid body simulator (Fig. 1, rigid_body) for optimizing robot controllers. This simulator supports rigid body collision and friction, spring forces, joints, and actuation. The simulation is end-to-end differentiable except for a countable number of discontinuities. Interestingly, although the forward simulator works well, naively differentiating it with DiffSim leads to completely misleading gradients, due to the rigid body collisions. We discuss the cause and solution of this issue below. Improving collision gradients Consider the rigid ball example in Fig. 4 (left), where a rigid ball collides with a friction-less ground. Gravity is ignored, and due to conservation of kinetic energy the ball keeps a constant speed even after this elastic collision. In the forward simulation, using a small ∆t often leads to a reasonable , as done in many physics simulators. Lowering the initial ball height will increase the final ball height, since there is less distance to travel before the ball hits the ground and more after (see the loss curves in Fig.4, middle right). However, using a naive time integrator, no matter how small ∆t is, the evaluated gradient of final height w.r.t. initial height will be 1 instead of −1. This counter-intuitive behavior is due to the fact that time discretization itself is not differentiated by the compiler. We propose a simple solution of adding continuous collision resolution (see, for example,), which considers precise time of impact (TOI), to the forward program (Fig. 4, middle left). Although it barely improves the forward simulation (Fig. 4, middle right), the gradient will be corrected effectively (Fig. 4, right). The details of continuous collision detection are in Appendix F. In real-world simulators, we find the TOI technique leads to significant improvement in gradient quality in controller optimization tasks (Fig. 5). Having TOI or not barely affects forward simulation: in the supplemental video, we show that a robot controller optimized in a simulator with TOI, actually works well in a simulator without TOI. The takeaway is, differentiating physical simulators does not always yield useful gradients of the physical system being simulated, even if the simulator does forward simulation well. In Appendix G, we discuss some additional gradient issues we have encountered. . However, physical simulation requires complex and customizable operations due to the intrinsic computational irregularity. Using the aforementioned frameworks, programmers have to compose these coarse-grained basic operations into desired complex operations. Doing so often leads to unsatisfactory performance. Earlier work on automatic differentiation focuses on transforming existing scalar code to obtain derivatives (e.g. , redner (a), Mitsuba 2 ) to learn from 3D scenes. We have presented DiffSim, a new differentiable programming language designed specifically for building high-performance differentiable physical simulators. Motivated by the need for supporting megakernels, imperative programming, and flexible indexing, we developed a tailored two-scale automatic differentiation system. We used DiffSim to build 10 simulators and integrated them into deep neural networks, which proved the performance and productivity of DiffSim over existing systems. We hope our programming language can greatly lower the barrier of future research on differentiable physical simulation in the machine learning and robotics communities. Workload differences between deep learning and differentiable physical simulation Existing differentiable programming tools for deep learning are typically centered around large data blobs. For example, in AlexNet, the second convolution layer has size 27 × 27 × 128 × 128. These tools usually provide users with both low-level operations such as tensor add and mul, and high-level operations such as convolution. The bottleneck of typical deep-learning-based computer vision tasks are convolutions, so the provided high-level operations, with very high arithmetic intensity 2, can fully exploit hardware capability. However, the provided operations are "atoms" of these differentiable programming tools, and cannot be further customized. Users often have to use low-level operations to compose their desired high-level operations. This introduces a lot of temporary buffers, and potentially excessive GPU kernel launches. As shown in Hu et al. (2019b), a pure TensorFlow implementation of a complex physical simulator is 132× slower than a CUDA implementation, due to excessive GPU kernel launches and the lack of producer-consumer locality 3. The table below compares DiffSim with existing tools for build differentiable physical simulators. Primal and adjoint kernels Recall that in DiffSim, (primal) kernels are operators that take as input multiple tensors (e.g., X, Y) and output another set of tensors. Mathematically, kernel f has the form Kernels usually execute uniform operations on these tensors. When it comes to differentiable programming, a loss function is defined on the final output tensors. The gradients of the loss function "L" with respect to each tensor are stored in adjoint tensors and computed via adjoint kernels. The adjoint tensor of (primal) tensor X ijk is denoted as X * ijk. Its entries are defined by X * ijk = ∂L/∂X ijk. At a high level, our automatic differentiation (AD) system transforms a primal kernel into its adjoint form. Mathematically, In this section we demonstrate how to use checkpointing via complex kernels. The goal of checkpointing is to use recomputation to save memory space. We demonstrate this using the diffmpm example, whose simulation cycle consists of particle to grid transform (p2g), grid boundary conditions (grid_op), and grid to particle transform (g2p). We assume the simulation has O(n) time steps. A naive implementation without checkpointing allocates O(n) copied of the simulation grid, which can cost a lot of memory space. Actually, if we recompute the grid states during the backward simulation time step by redoing p2g and grid_op, we can reused the grid states and allocate only one copy. This checkpointing optimization is demonstrated in the code below: Given a simulation with O(n) time steps, if all simulation steps are recorded, the space consumption is O(n). This linear space consumption is sometimes too large for high-resolution simulations with long time horizon. Fortunately, we can reduce the space consumption using a segment-wise checkpointing trick: We split the simulation into segments of S steps, and in forward simulation store only the first simulation state in each segment. During backpropagation when we need the remaining simulation states in a segment, we recompute them based on the first state in that segment. Note that if the segment size is O(S), then we only need to store O(n/S) simulation steps for checkpoints and O(S) reusable simulation steps for backpropagation within segments. The total space consumption is O(S + n/S). We follow and implemented a 3D differentiable liquid simulator. Our liquid simulation can be two-way coupled with elastic object simulation [diffmpm] (Figure 6, right). (a) smoke (b) wave Backpropagating Through Pressure Projection We followed the baseline implementation in Autograd, and used 10 Jacobi iterations for pressure projection. Technically, 10 Jacobi iterations are not sufficient to make the velocity field fully divergence-free. However, in this example, it does a decent job, and we are able to successfully backpropagate through the unrolled 10 Jacobi iterations. In larger-scale simulations, 10 Jacobi iterations are likely not sufficient. Assuming the Poisson solve is done by an iterative solver (e.g. multigrid preconditioned conjugate gradients, MGPCG) with 5 multigrid levels and 50 conjugate gradient iterations, then automatic differentiation will likely not be able to provide gradients with sufficient numerical accuracy across this long iterative process. The accuracy is likely worse with conjugate gradients present, as they are known to numerically drift as the number of iterations increases. In this case, the user can still use DiffSim to implement the forward MGPCG solver, while implementing the backward part of the Poisson solve manually, likely using adjoint methods . DiffSim provides "complex kernels" to override the built-in AD system, as shown in Appendix C. We adopt the wave equation in to model shallow water height field evolution: where u is the height of shallow water, c is the "speed of sound" and α is a damping coefficient. We use theu andü notations for the first and second order partial derivatives of u w.r.t time t respectively. used the finite different time-domain (FDTD) method (Larsson & Thomée, 2008) to discretize Eqn. 1, yielding an update scheme: We implemented this wave simulator in DiffSim to simulate shallow water. We used a grid of resolution 128 × 128 and 256 time steps. The loss function is defined as where T is the final time step, andû is the target height field. 200 gradient descent iterations are then used to optimize the initial height field. We setû to be the pattern "Taichi", and Fig. 7b shows the unoptimized and optimized wave evolution. We set the "Taichi" symbol as the target pattern. Fig. 7b shows the unoptimized and optimized final wave patterns. More details on discretization is in Appendix E. We extend the mass-spring system in the main text with ground collision and a NN controller. The optimization goal is to maximize the distance moved forward with 2048 time steps. We designed three mass-spring robots as shown in Fig. 8 (left). A differentiable rigid body simulator is built for optimizing a billiards strategy (Fig. 8, middle). We used forward Euler for the billiard ball motion and conservation of momentum and kinetic energy for collision resolution. E.7 DIFFERENTIABLE RIGID BODY SIMULATOR [rigid_body] Are rigid body collisions differentiable? It is worth noting that discontinuities can happen in rigid body collisions, and at a countable number of discontinuities the objective function is nondifferentiable. However, apart from these discontinuities, the process is still differentiable almost everywhere. The situation of rigid body collision is somewhat similar to the "ReLU" activation function in neural networks: at point x = 0, ReLU is not differentiable (although continuous), yet it is still widely adopted. The rigid body simulation cases are more complex than ReLU, as we have not only non-differentiable points, but also discontinuous points. Based on our experiments, in these impulse-based rigid body simulators, we still find the gradients useful for optimization despite the discontinuities, especially with our time-of-impact fix. We implemented differentiable renderers to visualize the refracting water surfaces from wave. We use finite differences to reconstruct the water surface models based on the input height field and refract camera rays to sample the images, using bilinear interpolation for meaningful gradients. To show our system works well with other differentiable programming systems, we use an adversarial optimization goal: fool VGG-16 into thinking that the refracted squirrel image is a goldfish (Fig. 9). E.9 DIFFERENTIABLE VOLUME RENDERER [volume_renderer] We implemented a basic volume renderer that simply uses ray marching (we ignore light, scattering, etc.) to integrate a density field over each camera ray. In this task, we render a number of target images from different viewpoints, with the camera rotated around the given volume. The goal is then to optimize for the density field of the volume that would produce these target images: we render candidate images from the same viewpoints and compute an L2 loss between them and the target images, before performing gradient descent on the density field (Fig. 10). Recall Coulomb's law: F = k q1q2 r 2r. In the right figure, there are eight electrodes carrying static charge. The red ball also carries static charge. The controller, which is a two-layer neural network, tries to manipulate the electrodes so that the red ball follows the path of the blue ball. The bigger the electrode, the more positive charge it carries. # Note that with time of impact, dt is divided into two parts, # the first part using old_v, and second part using new_v new_x = old_x + toi * old_v + (dt -toi) * new_v In rigid body simulation, the implementation follows the same idea yet is slightly more complex. Please refer to rigid_body.py for more details. Initialization matters: flat lands and local minima in physical processes A trivial example of objective flat land is in billiards. Without proper initialization, gradient descent will make no progress since gradients are zero (Fig. 11). Also note the local minimum near (−5, 0.03). In mass_spring and rigid_body, once the robot falls down, gradient descent will quickly become trapped. A robot on the ground will make no further progress, no matter how it changes its controller. This leads to a more non-trivial local minimum and zero gradient case. Ideal physical models are only "ideal": discontinuities and singularities Real-world macroscopic physical processes are usually continuous. However, building upon ideal physical models, even in the forward physical simulation can contain discontinuities. For example, in a rigid body model with friction, changing the initial rotation of the box can lead to different corners hitting the ground first, and in a discontinuity (Fig. 12). In electric and mass_spring, due to the 1 r 2 and 1 r terms, when r → 0, gradients can be very inaccurate due to numerical precision issues. Note that d(1/r)/dr = −1/r 2, and the gradient is more numerically problematic than the primal for a small r. Safeguarding r is critically important for gradient stability. Figure 12: Friction in rigid body with collision is a common source of discontinuity. In this scene a rigid body hits the ground. Slightly rotating the rigid body changes which corner (A/B) hits the ground first, and different normal/friction impulses will be applied to the rigid body. This leads to a discontinuity in its final position (loss=final y coordinate). [Reproduce: python3 rigid_body_discontinuity.py] Please see our supplemental video for more details.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We study the problem of learning and optimizing through physical simulations via differentiable programming, using our proposed DiffSim programming language and compiler.

Represent the TLDR for finding the corresponding scientific article

We propose a simple, general, and space-efficient data format to accelerate deep learning training by allowing sample fidelity to be dynamically selected at training time


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Local explanation frameworks aim to rationalize particular decisions made by a black-box prediction model. Existing techniques are often restricted to a specific type of predictor or based on input saliency, which may be undesirably sensitive to factors unrelated to the model's decision making process. We instead propose sufficient input subsets that identify minimal subsets of features whose observed values alone suffice for the same decision to be reached, even if all other input feature values are missing. General principles that globally govern a model's decision-making can also be revealed by searching for clusters of such input patterns across many data points. Our approach is conceptually straightforward, entirely model-agnostic, simply implemented using instance-wise backward selection, and able to produce more concise rationales than existing techniques. We demonstrate the utility of our interpretation method on neural network models trained on text and image data. The rise of neural networks and nonparametric methods in machine learning (ML) has driven significant improvements in prediction capabilities, while simultaneously earning the field a reputation of producing complex black-box models. Vital applications, which could benefit most from improved prediction, are often deemed too sensitive for opaque learning systems. Consider the widespread use of ML for screening people, including models that deny defendants' bail or reject loan applicants. It is imperative that such decisions can be interpretably rationalized. Interpretability is also crucial in scientific applications, where it is hoped that general principles may be extracted from accurate predictive models.One simple explanation for why a particular black-box decision is reached may be obtained via a sparse subset of the input features whose values form the basis for the model's decision -a rationale. For text (or image) data, a rationale might consist of a subset of positions in the document (or image) together with the words (or pixel-values) occurring at these positions (see FIG0 . To ensure interpretations remain fully faithful to an arbitrary model, our rationales do not attempt to summarize the (potentially complex) operations carried out within the model, and instead merely point to the relevant information it uses to arrive at a decision. For high-dimensional inputs, sparsity of the rationale is imperative for greater interpretability. Here, we propose a local explanation framework to produce rationales for a learned model that has been trained to map inputs x P X via some arbitrary learned function f: X Ñ R. Unlike many other interpretability techniques, our approach is not restricted to vector-valued data and does not require gradients of f. Rather, each input example is solely presumed to have a set of indexable features x " rx 1,..., x p s, where each x i P R d for i P rps " t1,..., pu. We allow for features that are unordered (set-valued input) and whose number p may vary from input to input. A rationale corresponds to a sparse subset of these indices S Ď rps together with the specific values of the features in this subset. To understand why a certain decision was made for a given input example x, we propose a particular rationale called a sufficient input subset (SIS). Each SIS consists of a minimal input pattern present in x that alone suffices for f to produce the same decision, even if provided no other information about the rest of x. Presuming the decision is based on f pxq exceeding some pre-specified threshold τ P R, we specifically seek a minimal-cardinality subset S of the input features such that f px S q ě τ. Throughout, we use x S P X to denote a modified input example in which all information about the values of features outside subset S has been masked with features in S remaining at their original values. Thus, each SIS characterizes a particular standalone input pattern that drives the model toward this decision, providing sufficient justification for this choice from the model's perspective, even without any information on the values of the other features in x. In classification settings, f might represent the predicted probability of class C where we decide to assign the input to class C if f pxq ě τ, chosen based on precision/recall considerations. Each SIS in such an application corresponds to a small input pattern that on its own is highly indicative of class C, according to our model. Note that by suitably defining f and τ with respect to the predictor outputs, any particular decision for input x can be precisely identified with the occurrence of f pxq ě τ, where higher values of f are associated with greater confidence in this decision. For a given input x where f pxq ě τ, this work presents a simple method to find a complete collection of sufficient input subsets, each satisfying f px S q ě τ, such that there exists no additional SIS outside of this collection. Each SIS may be understood as a disjoint piece of evidence that would lead the model to the same decision, and why this decision was reached for x can be unequivocally attributed to the SIS-collection. Furthermore, global insight on the general principles underlying the model's decision-making process may be gleaned by clustering the types of SIS extracted across different data points (see FIG4 and TAB0). Such insights allow us to compare models based not only on their accuracy, but also on human-determined relevance of the concepts they target. Our method's simplicity facilitates its utilization by non-experts who may know very little about the models they wish to interrogate. Certain neural network variants such as attention mechanisms and the generator-encoder of have been proposed as powerful yet human-interpretable learners. Other interpretability efforts have tailored decompositions to certain convolutional/recurrent networks, but these approaches are model-specific and only suited for ML experts. Many applications necessitate a model outside of these families, either to ensure supreme accuracy, or if training is done separately with access restricted to a black-box API.An alternative model-agnostic approach to interpretability produces local explanations of f for a particular input x (e.g. an individual classification decision). Popular local explanation techniques produce attribution scores that quantify the importance of each feature in determining the output of f at x. Examples include LIME, which locally approximates f, saliency maps based on f -gradients, Layer-wise Relevance Propagation, as well as the discrete DeepLIFT approach and its continuous variant -Integrated Gradients (IG), developed to ensure attributions reflect the cumulative difference in f at x vs. a reference input. A separate class of input-signal-based explanation techniques such as DeConvNet, Guided Backprop, and PatternNet employ gradients of f in order to identify input patterns that cause f to output large values. However, many such gradient-based saliency methods have been found unreliable, depending not only on the learned function f, but also on its specific architectural implementation and how inputs are scaled. More similar to our approach are recent techniques which also aim to identify input patterns that best explain certain decisions, but additionally require either a predefined set of such patterns or an auxiliary neural network trained to identify them. In comparison with the aforementioned methods, our SIS approach presented here is conceptually simple, completely faithful to any type of model, requires no access to gradients of f, requires no additional training of the underlying model f, and does not require training any auxiliary explanation model. Also related to our subset-selection methodology are the ideas of Li et al. and Fong & Veldadi, which for a particular input example aim to identify a minimal subset of features whose deletion causes a substantial drop in f such that a different decision would be reached. However, this objective can undesirably produce adversarial artifacts that are not easy to interpret. In contrast, we focus on identifying disjoint minimal subsets of input features whose values suffice to ensure f outputs significantly positive predictions, even in the absence of any other information about the rest of the input. While the techniques used in produce rationales that remain strongly dependent on the rest of the input outside of the selected feature subset, each rationale revealed by our SIS approach is independently considered by f as an entirely sufficient justification for a particular decision in the absence of other information. Our approach to rationalizing why a particular black-box decision is reached only applies to input examples x P X that meet the decision criterion f pxq ě τ. For such an input x, we aim to identify a SIS-collection of disjoint feature subsets S 1,..., S K Ď rps that satisfy the following criteria: f px S k q ě τ for each k " 1,..., K There exists no feature subset S 1 Ă S k for some k " 1,..., K such that f px S 1 q ě τ f px R q ă τ for R " rps z Ť K k"1 S k (the remaining features outside of the SIS-collection) Criterion ensures that for any SIS S k, the values of the features in this subset alone suffice to justify the decision in the absence of any information regarding the values of the other features. To ensure information that is not vital to reach the decision is not included within the SIS, criterion encourages each SIS to contain a minimal number of features, which facilitates interpretability. Finally, we require that our SIS-collection satisfies a notion of completeness via criterion, which states that the same decision is no longer reached for the input after the entire SIS-collection has been masked. This implies the remaining feature values of the input no longer contain sufficient evidence for the same decision. FIG1 show SIS-collections found in text/image inputs. Recall that x S P X denotes a modified input in which the information about the values of features outside subset S is considered to be missing. We construct x S as new input whose values on features in S are identical to those in the original x, and whose remaining features x i P rpszS are each replaced by a special mask z i P R di used to represent a missing observation. While certain models are specially adapted to handle inputs with missing observations, this is generally not the case. To ensure our approach is applicable to all models, we draw inspiration from data imputation techniques which are a common way to represent missing data.Two popular strategies include hot-deck imputation, in which unobserved values are sampled from their marginal feature distribution, and mean imputation, in which each z i simply fixed to the average value of feature i in the data. Note that for a linear model, these two strategies are expected to produce an identical change in prediction f pxq´f px S q. We find in practice that the change in predictions ing from either masking strategy is roughly equivalent even for nonlinear models such as neural networks (FIG0). In this work, we favor the mean-imputation approach over samplingbased imputation, which would be computationally-expensive and nondeterministic (undesirable for facilitating interpretability). One may also view z as the baseline input value used by feature attribution methods, a value which should not lead to particularly noteworthy decisions. Since our interests primarily lie in rationalizing atypical decisions, the average input arising from mean imputation serves as a suitable baseline. Zeros have also been used to mask image/categorical data, but empirically, this mask appears undesirably more informative than the mean (predictions more affected by zero-masking).For an arbitrarily complex function f over inputs with many features p, the combinatorial search to identify sets which satisfy objectives- is computationally infeasible. To find a SIS-collection in DISPLAYFORM0 Update S Ð S Y tiu 5 if f px S q ě τ: return S 6 else: return None practice, we employ a straightforward backward selection strategy, which is here applied separately on an example-by-example basis (unlike standard statistical tools which perform backward selection globally to find a fixed set of features for all inputs). The SIScollection algorithm details our straightforward procedure to identify disjoint SIS subsets that satisfy- approximately (as detailed in §3.1) for an input x P X where f pxq ě τ.Our overall strategy is to find a SIS subset S k (via BackSelect and FindSIS), mask it out, and then repeat these two steps restricting each search for the next SIS solely to features disjoint from the currently found SIS-collection S 1,..., S k, until the decision of interest is no longer supported by the remaining feature values. In the BackSelect procedure, S Ă rps denotes the set of remaining unmasked features that are to be considered during backward selection. For the current subset S, step 3 in BackSelect identifies which remaining feature i P S produces the minimal reduction in f px S q´f px Sztiu q (meaning it least reduces the output of f if additionally masked), a question trivially answered by running each of the remaining possibilities through the model. This strategy aims to gradually mask out the least important features in order to reveal the core input pattern that is perceived by the model as sufficient evidence for its decision. Finally, we build our SIS up from the last features omitted during the backward selection, selecting a value just large enough to meet our sufficiency criterion. Because this approach always queries a prediction over the joint set of remaining features S, it is better suited to account for interactions between these features and ensure their sufficiency (i.e. that f px S q ě τ) compared to a forward selection in the opposite direction which builds the SIS upwards one feature at a time by greedily maximizing marginal gains. Throughout its execution, BackSelect attempts to maintain the sufficiency of x S as the set S shrinks. Given p input features, our algorithm requires Opp 2 kq evaluations of f to identify k SIS, but we can achieve Oppkq by parallelizing each argmax in BackSelect (e.g. batching on GPU). Throughout, let S 1,..., S K denote the output of SIScollection when applied to a given input x for which f pxq ě τ. Disjointness of these sets is crucial to ensure computational tractability and that the number of SIS per example does not grow huge and hard to interpret. Proposition 1 below proves that each SIS produced by our procedure will satisfy an approximate notion of minimality. Because we desire minimality of the SIS as specified by, it is not appropriate to terminate the backward elimination in BackSelect as soon as the sufficiency condition f px S q ě τ is violated, due to the possible presence of local minima in f along the path of subsets encountered during backward selection (as shown in FIG1).Proposition 2 additionally guarantees that masking out the entirety of the feature values in the SIScollection will ensure the model makes a different decision. Given f pxq ě τ, it is thus necessarily the case that the observed values responsible for this decision lie within the SIS-collection S 1,..., S K. We point out that for an easily reached decision, where f pzq ě τ (i.e. this decision is reached even for the average input), our approach will not output any SIS. Because this same decision would likely be anyway reached for a vast number of inputs in the training data (as a sort of default decision), it is conceptually difficult to grasp what particular aspect of the given x is responsible. Proposition 1. There exists no feature i in any set S 1,..., S K that can be additionally masked while retaining sufficiency of the ing subset (i.e. f px S k ztiu q ă τ for any k " 1, . . ., K, i P S k). Also, among all subsets S considered during the backward selection phase used to produce S k, this set has the smallest cardinality of those which satisfy f px S q ě τ. Proposition 2. For x rpszS˚, modified by masking all features in the entire SIS-collection S˚" Ť K k"1 S k, we must have: f px rpszS˚q ă τ when S˚‰ rps. Unfortunately, nice assumptions like convexity/submodularity are inappropriate for estimated functions in ML. We present various simple forms of practical decision functions for which our algorithms are guaranteed to produce desirable explanations. Example 1 considers interpreting functions of a generalized linear form, Examples 2 & 3 describe functions whose operations resemble generalized logical OR & AND gates, and Example 4 considers functions that seek out a particular input pattern. Note that features ignored by f are always masked in our backward selection and thus never appear in the ing SIS-collection. Example 1. Suppose the input data are vectors and f pxq " gpβ T x`β 0 q, where g is monotonically increasing. We also presume τ ą gpβ 0 q and the data were centered such that each feature has mean zero (for ease of notation). In this case, S 1,..., S K must satisfy criteria-. S 1 will consist of the features whose indices correspond to the largest entries of tβ 1 x 1,..., β p x p u for some suitable that depends on the value of τ. It is also guaranteed that f px S1 q ě f px S q for any subset S Ď rps of the same cardinality |S| ". For each individual feature i where gpβ i x i`β0 q ě τ, there will be exist a corresponding SIS S k consisting only of tiu. No SIS will include features whose coefficient β i " 0, or those whose difference between the observed and average value z i (" 0 here) is of an opposite sign than the corresponding model coefficient (i.e. β i px i´zi q ď 0). DISPLAYFORM0.., g L, such that for the given x and threshold τ: DISPLAYFORM1 Such f might be functions that model strong interactions between the features in each S k or look for highly specific value patterns to occur these subsets. In this case, SIScollection will return L sets such that DISPLAYFORM2 qu and the same conditions from Example 2 are met, then SIScollection will return a single set DISPLAYFORM3 p with f pxq " hp||x S´cS ||q where h is monotonically decreasing and c S specifies a fixed pattern of input values for features in a certain subset S. For input x and threshold choice τ " f pxq, SIScollection will return a single set S 1 " ti P S: |x i´ci | ă |z i´ci |u. We apply our methods to analyze neural networks for text and image data. SIScollection is compared with alternative subset-selection methods for producing rationales (see descriptions in Supplement §S1). Note that our BackSelect procedure determines an ordering of elements, R, subsequently used to construct the SIS. Depictions of each SIS are shaded based on the feature order in R (darker = later), which can indicate relative feature importance within the SIS. In the "Suff. IG," "Suff. LIME," and "Suff. Perturb." (sufficiency constrained) methods, we instead compute the ordering of elements R according to the feature attribution values output by integrated gradients, LIME, or a perturbative approach that measures the change in prediction when individually masking each feature (see §S1). The rationale subset S produced under each method is subsequently assembled using FindSIS exactly as in our approach and thus is guaranteed to satisfy f px S q ě τ. In the "IG," "LIME," and "Perturb." (length constrained) methods, we use the same previously described ordering R, but always select the same number of features in the rationale as in the SIS produced by our method (per example). We first consider a dataset of beer reviews from BeerAdvocate. Taking the text of a review as input, different LSTM networks are trained to predict user-provided numerical ratings of aspects like aroma, appearance, and palate (details in §S2). FIG0 shows a sample beer review where we highlight the SIS identified for the LSTM that predicts each aspect. Each SIS only captures sentiment toward the relevant aspect. FIG1 depicts the SIS-collection identified from a review the LSTM decided to flag for positive aroma. FIG2 shows that when the alternative methods described in §4 are length constrained, the rationales they produce often badly fail to meet our sufficiency criterion. Thus, even though the same number of feature values are preserved in the rationale and these alternative methods select the features to which they have assigned the largest attribution values, their rationales lead to significantly reduced f outputs compared to our SIS subsets. If the sufficiency constraint is instead enforced for these alternative methods, the rationales they identify become significantly larger than those produced by SIScollection, and also contain many more unimportant features (Table S2, FIG1).Benchmarking interpretability methods is difficult because a learned f may behave counterintuitively such that seemingly unreasonable model explanations are in fact faithful descriptions of a model's decision-making process. For some reviews, a human annotator has manually selected which sentences carry the relevant sentiment for the aspect of interest, so we treat these annotations as an alternative rationale for the LSTM prediction. For a review x whose true and predicted aroma exceed our decision threshold, we define the quality of human-selected sentences for model explanation QHS " f px S q´f pxq where S is the human-selected-subset of words in the review (see examples in FIG6). High variability of QHS in the annotated reviews FIG3 indicates the human rationales often do not contain sufficient information to preserve the LSTM's decision. FIG3 shows the LSTM makes many decisions based on different subsets of the text than the parts that humans find appropriate for this task. Reassuringly, our SIS more often lie within the selected annotation for reviews with high QHS scores. We also study a 10-way CNN classifier trained on the MNIST handwritten digits data. Here, we only consider predicted probabilities for one class of interest at a time and always set τ " 0.7 as the probability threshold for deciding that an image belongs to the class. We extract the SIS-collection from all corresponding test set examples (details in §S3). Example images and corresponding SIScollections are shown in Figures 6, 7, and S27. FIG5 illustrates how the SIS-collection drastically changes for an example of a correctly-classified 9 that has been adversarially manipulated to become confidently classified as the digit 4. Furthermore, these SIS-collections immediately enable us to understand why certain misclassifications occur FIG5 ). Identifying the different input patterns that justify a decision can help us better grasp the general operating principles of a model. To this end, we cluster all of the SIS produced by SIScollection applied across a large number of data examples that received the same decision. Clustering is done via DBSCAN, a widely applicable algorithm that merely requires specifying pairwise distances between points.We first apply this procedure to the SIS found across all held-out beer reviews (Test-Fold in TAB0) that received positive aroma predictions from our LSTM network. The distance between two SIS is taken as the Jaccard distance between their bag of words representations. Three clusters depicted in TAB0 ) reveal isolated phrases that the LSTM associates with positive aromas in the absence of other context. We also apply DBSCAN clustering to the SIS found across all MNIST test-examples confidently identified by the CNN as a particular class. Pairwise distances are here defined as the energy distance over pixel locations between two SIS subsets (see §S3.3). FIG4 depicts the SIS clusters identified for digit 4 (others in FIG1). These reveal distinct feature patterns learned by the CNN to distinguish 4 from other digits, which are clearly present in the vast majority of test set images confidently classified as a 4. For example, cluster C 8 depicts parallel slanted lines, a pattern that never occurs in other digits. The general insights revealed by our SIS-clustering can also be used to compare the operatingbehavior of different models. For the beer reviews, we also train a CNN to compare with our existing LSTM (see §S2.6). For MNIST, we train a multilayer perceptron (MLP) and compare to our existing CNN (see §S3.5). Both networks exhibit similar performance in each task, so it is not immediately clear which model would be preferable in practice. FIG8 shows the SIS extracted under one model are typically insufficient to receive the same decision from the other model, indicating these models base their positive predictions on different evidence. TAB1 contains of jointly clustering the SIS extracted from beer reviews with positive aroma predictions under our LSTM or text-CNN. This CNN tends to learn localized (unigram/bigram) word patterns, while the LSTM identifies more complex multi-word interactions that truly seem more relevant to the target aroma value. Many CNN-SIS are simply phrases with universally-positive sentiment, indicating this model is less capable at distinguishing between positive sentiment toward This work introduced the idea of interpreting black-box decisions on the basis of sufficient input subsets -minimal input patterns that alone provide sufficient evidence to justify a particular decision. Our methodology is easy to understand for non-experts, applicable to all ML models without any additional training steps, and remains fully faithful to the underlying model without making approximations. While we focus on local explanations of a single decision, clustering the SISpatterns extracted from many data points reveals insights about a model's general decision-making process. Given multiple models of comparable accuracy, SIS-clustering can uncover critical operating differences, such as which model is more susceptible to spurious training data correlations or will generalize worse to counterfactual inputs that lie outside the data distribution. Kleinberg J, Lakkaraju H, Leskovec J, Ludwig J, Mullainathan S Human decisions and machine predictions. The Quarterly Journal of Economics 133: 237-293. Sirignano JA, Sadhwani A, Giesecke K Deep learning for mortgage risk. arXiv:160702470. Doshi-Velez F, Kim B Towards a rigorous science of interpretable machine learning. arXiv:170208608. The mythos of model interpretability. In: ICML Workshop on Human Interpretability of Machine Learning. Shrikumar A, Greenside P, Kundaje A Learning important features through propagating activation differences. In: International Conference on Machine Learning. Lei T, Barzilay R, Jaakkola T Rationalizing neural predictions. In: Empirical Methods in Natural Language Processing. Ester M, Kriegel HP, Sander J, Xu X A density-based algorithm for discovering clusters a density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining.[List of TAB0 In Section 3, we describe a number of alternative methods for identifying rationales for comparison with our method. We use methods based on integrated gradients BID0, LIME BID1, and feature perturbation. Note that integrated gradients is an attribution method which assigns a numerical score to each input feature. LIME likewise assigns a weight to each feature using a local linear regression model for f around x. In the perturbative approach, we compute the change in prediction when each feature is individually masked, as in Equation 1 (of Section S2.4). Each of these feature orderings R is used to construct a rationale using the FindSIS procedure (Section 3) for the "Suff. IG," "Suff. LIME," and "Suff. Perturb." (sufficiency constrained) methods. Note that our text classification architecture (described in Section S2.2) encodes discrete words as 100-dimensional continuous word embeddings. The integrated gradients method returns attribution scores for each coordinate of each word embedding. For each word embedding x i P x (where each x i P R 100), we summarize the attributions along the corresponding embedding into a single score y i using the L 1 norm: y i " ř d |x id | and compute the ordering R by sorting the y i values. We use an implementation of integrated gradients for Keras-based models from https://github. com/hiranumn/IntegratedGradients. In the case of the beer review dataset (Section 4.1), we use the mean embedding vector as a baseline for computing integrated gradients. As suggested in BID0, we verified that the prediction at the baseline and the integrated gradients sum to approximately the prediction of the input. For LIME and our beer reviews dataset, we use the approach described in BID1 for textual data, where individual words are removed entirely from the input sequence. We use the implementation of LIME at: https://github.com/marcotcr/lime. The LimeTextExplainer module is used with default parameters, except we set the maximal number of features used in the regression to be the full input length so we can order all input features. Additionally, we explore methods in which we use the same ordering R by these alternative methods but select the same number of input features in the rationale to be the median SIS length in the SIS-collection computed by our method on each example: the "IG," "LIME," and "Perturb. " (length constrained) methods. We compute the feature ordering based on the absolute value of the non-zero integrated gradient attributions. Note that for the length constrained methods, there is no guarantee of sufficiency f px S q ě τ for any input subset S. As done in BID2, we use a preprocessed version of the BeerAdvocate 2 dataset 3 which contains decorrelated numerical ratings toward three aspects: aroma, appearance, and palate (each normalized to r0, 1s). Dataset statistics can be found in TAB0. Reviews were tokenized by converting to lowercase and filtering punctuation, and we used a vocabulary containing the top 10,000 most common words. The data also contain subset of human-annotated reviews, in which humans manually selected full sentences in each review that describe the relevant aspects BID3. This annotated set was never seen during training and used solely as part of our evaluation. Long short-term memory (LSTM) networks are commonly employed for natural language tasks such as sentiment analysis BID4 BID5. We use a recurrent neural network (RNN) architecture with two stacked LSTMs as follows:1. Input/Embeddings Layer: Sequence with 500 timesteps, the word at each timestep is represented by a (learned) 100-dimensional embedding 2. LSTM Layer 1: 200-unit recurrent layer with LSTM (forward direction only) 3. LSTM Layer 2: 200-unit recurrent layer with LSTM (forward direction only) 4. Dense: 1 neuron (sentiment output), sigmoid activationWith this architecture, we use the Adam optimizer BID6 to minimize mean squared error (MSE) on the training set. We use a held-out set of 3,000 examples for validation (sampled at random from the pre-defined test set used in BID2). Our test set consists of the remaining 7,000 test examples. Training are shown in TAB0. In Section 3, we discuss the problem of masking input features. Here, we show that the meanimputation approach (in which missing inputs are masked with a mean embedding, taken over the entire vocabulary) produces a nearly identical change in prediction to a nondeterministic hot-deck approach (in which missing inputs are replaced by randomly sampling feature-values from the data). FIG0 shows the change in prediction f pxztiuq´f pxq by both imputation techniques after drawing a training example x and word x i P x (both uniformly at random) and replacing x i with either the mean embedding or a randomly selected word (drawn from the vocabulary, based on counts in the training corpus). This procedure is repeated 10,000 times. Both ing distributions have mean near zero (µ mean-embedding "´7.0e´4, µ hot-deck "´7.4e´4), and the distribution for mean embedding is slightly narrower (σ mean-embedding " 0.013, σ hot-deck " 0.018). We conclude that mean-imputation is a suitable method for masking information about particular feature values in our SIS analysis. We also explored other options for masking word information, e.g. replacement with a zero embedding, replacement with the learned <PAD> embedding, and simply removing the word entirely from the input sequence, but each of these alternative options led to undesirably larger changes in predicted values as a of masking, indicating they appear more informative to f than replacement via the feature-mean. For each feature i in the input sequence, we quantify its marginal importance by individually perturbing only this feature:Feature Importancepiq " prediction on original input´prediction with feature i masked FIG0: Change in prediction (f pxztiuq´f pxq) after masking a randomly chosen word with mean imputation or hot-deck imputation. 10,000 replacements were sampled from the aroma beer reviews training set. Note that these marginal Feature Importance scores are identical to those of the Perturb. method described in Section S1. The marginal Feature Importance scores are summarized in TAB1 and FIG1. Compared to the Suff. IG and Suff. LIME methods, our SIScollection technique produces rationales that are much shorter and contain fewer irrelevant (i.e. not marginally important) features TAB1, FIG1. Note that by construction, the rationales of the Suff. Perturb. method contain features with the greatest Feature Importance, since this precisely how the ranking in Suff. Perturb. is defined. We apply our method to the set of reviews containing sentence-level annotations. Note that these reviews (and the human annotations) were not seen during training. We choose thresholds τ`" 0.85, FIG1: Importance of individual features in the rationales for aroma prediction in beer reviews FIG2: Length of rationales for aroma prediction FIG3: Predictive distribution on the annotation set (held-out) using the LSTM model for aroma. Vertical lines indicate decision thresholds (τ`" 0.85, τ´" 0.45) selected for SIScollection. τ´" 0.45 for strong positive and strong negative sentiment, respectively, and extract the complete set of sufficient input subsets using our method. Note that in our formulation above, we apply our method to inputs x where f pxq ě τ. For the sentiment analysis task, we analogously apply our method for both f pxq ě τ`and´f pxq ě´τ´, where the model predicts either strong positive or strong negative sentiment, respectively. These thresholds were set empirically such that they were sufficiently apart, based on the distribution of predictions (FIG3). For most reviews, SIScollection outputs just one or two SIS sets (FIG4).We analyzed the predictor output following the elimination of each feature in the BackSelect procedure (Section 3). FIG5 shows the LSTM output on the remaining unmasked text f px Szti˚u q at each iteration of BackSelect, for all examples. This figure reveals that only a small number of features are needed by the model in order to make a strong prediction (most features can be removed without changing the prediction). We see that as those final, critical features are removed, there is a rapid, monotonic decrease in output values. Finally, we see that the first features to be removed by BackSelect are those which generally provide negative evidence against the decision. We demonstrate how our SIS-clustering procedure can be used to understand differences in the types of concepts considered important by different neural network architectures. In addition to the LSTM (see Section S2.2), we trained a convolutional neural network (CNN) on the same sentiment analysis task (on the aroma aspect). The CNN architecture is as follows: FIG6: Beer reviews (aroma) in which human-selected sentences (underlined) are aligned well (top) and poorly (bottom) with predictive model. Fraction of SIS in the human sentences corresponds accordingly. In the bottom example (poor alignment between human-selection and predictive model), our procedure has surfaced a case where the LSTM has learned features that diverge from what a human would expect (and may suggest overfitting). 1. Input/Embeddings Layer: Sequence with 500 timesteps, the word at each timestep is represented by a (learned) 100-dimensional embedding 2. Convolutional Layer 1: Applies 128 filters of window size 3 over the sequence, with ReLU activation 3. Max Pooling Layer 1: Max-over-time pooling, followed by flattening, to produce a p128, q representation 4. Dense: 1 neuron (sentiment output), sigmoid activation Note that a new set of embeddings was learned with the CNN. As with the LSTM model, we use Adam BID6 to minimize MSE on the training set. For the aroma aspect, this CNN achieves 0.016 (0.850), 0.025 (0.748), 0.026 (0.741), 0.014 (0.662) MSE (and Pearson ρ) on the Train, Validation, Test, and Annotation sets, respectively. We note that this performance is very similar to that from the LSTM (see TAB0).We apply our procedure to extract the SIS-collection from all applicable test examples using the CNN, as in Section 4.1. FIG8 shows the predictions from one model (LSTM or CNN) when fed input examples that are SIS extracted with respect to the other model (for reviews predicted to have positive sentiment toward the aroma aspect). For example, in FIG8, "CNN SIS Preds by LSTM" refers to predictions made by the LSTM on the set of sufficient input subsets produced by applying our SIScollection procedure on all examples x P X test for which f CNN pxq ě τ`. 4 Since the word embeddings are model-specific, we embed each SIS using the embeddings of the model making the prediction (note that while the embeddings are different, the vocabulary is the same across the models).In TAB1, we show five example clusters (and cluster composition) ing from clustering the combined set of all sufficient input subsets extracted by the LSTM and CNN on reviews in the test set for which a model predicts positive sentiment toward the aroma aspect. The complete clustering on reviews receiving positive sentiment predictions is shown in TAB7 for reviews receiving negative sentiment predictions. DISPLAYFORM0 For posterity, we include here from repeating the analysis in our paper for the two other non-aroma aspects measured in the beer reviews data: appearance and palate. FIG7: Change in appearance prediction (f pxztiuq´f pxq) after masking a randomly chosen word with mean imputation or hot-deck imputation. 10,000 replacements were sampled from the appearance beer reviews training set. FIG0: Change in palate prediction (f pxztiuq´f pxq) after masking a randomly chosen word with mean imputation or hot-deck imputation. 10,000 replacements were sampled from the palate beer reviews training set. Figure S19: Length of rationales for palate prediction FIG1: Importance of individual features in beer review palate rationales S3 Details of the MNIST Analysis The MNIST database of handwritten digits contains 60k training images and 10k test images BID7. All images are 28x28 grayscale, and we normalize them such that all pixel values are between 0 and 1. We use the convolutional architecture provided in the Keras MNIST CNN example. 5 The architecture is as follows:1. Input: (28 x 28 x 1) image, all values P r0, 1s 2. Convolutional Layer 1: Applies 32 3x3 filters with ReLU activation 3. Convolutional Layer 2: Applies 64 3x3 filters, with ReLU activation 4. Pooling Layer 1: Performs max pooling with a 2x2 filter and dropout probability 0.25 5. Dense Layer 1: 128 neurons, with ReLU activation and dropout probability 0.5 6. Dense Layer 2: 10 neurons (one per digit class), with softmax activation The Adadelta optimizer BID8 is used to minimize cross-entropy loss on the training set. The final model achieves 99.7% accuracy on the train set and 99.1% accuracy on the held-out test set. Original image (class 9). (c) SIS if backward selection were to terminate the first time prediction on remaining image drops below 0.7, corresponding to point C in (a) (CNN predicts class 9 with probability 0.700 on this SIS). (d) Actual SIS produced by our FindSIS algorithm, corresponding to point D in (a) (CNN predicts class 9 with probability 0.704 on this SIS). FIG1 demonstrates an example MNIST digit for which there exists a local minimum in the backward selection phase of our algorithm to identify the initial SIS. Note that if we were to terminate the backward selection as soon as predictions drop below the decision threshold, the ing SIS would be overly large, violating our minimality criterion. It is also evident from FIG1 that the smaller-cardinality SIS in (d), found after the initial local optimum in (c), presents a more interpretable input pattern that enables better understanding of the core motifs influencing our classifier's decisions. To avoid suboptimal , it is important to run a complete backward selection sweep until the entire input is masked before building the SIS upward, as done in our SIScollection procedure. To cluster SIS from the image data, we compute the pairwise distance between two SIS subsets S 1 and S 2 as the energy distance BID9 between two distributions over the image pixel coordinates that comprise the SIS, X 1 and X 2 P R 2: DISPLAYFORM0 Here, X i is uniformly distributed over the pixels that are selected as part of the SIS subset S i, X 1 i is an i.i.d. copy of X i, and ||¨|| represents the Euclidean norm. Unlike a Euclidean distance between images, our usage of the energy distance takes into account distances between the similar pixel coordinates that comprise each SIS. The energy distance offers a more efficiently computable integral probability metric than the optimal transport distance, which has been widely adopted as an appropriate measure of distance between images. We set the threshold τ " 0.7 for SIS to ensure that the model is confident in its class prediction (probability of the predicted class is ě 0.7). Almost all test examples initially have f pxq ě τ for the top class (FIG1). We identify all test examples that satisfy this condition and use SIS to identify all sufficient input subsets. The number of sufficient input subsets per digit is shown in FIG1.We apply our SIScollection algorithm to identify sufficient input subsets on MNIST test digits (Section 4.2). Examples of the complete SIS-collection corresponding to randomly chosen digits are shown in FIG1. We also cluster all the sufficient input subsets identified for each class (Section 4.3), depicting the in FIG1.In FIG5, we show an MNIST image of the digit 9, adversarially perturbed to 4, and the sufficient subsets corresponding to the adversarial prediction. Although a visual inspection of the perturbed image does not really reveal exactly how it has been manipulated, it becomes immediately clear from the SIS-collection for the adversarial image. These sets shows that the perturbation modifies pixels in such a way that input patterns similar to the typical SIS-collection for a 4 (FIG4) become embedded in the image. The adversarial manipulation was done using the Carlini-Wagner L 2 (CW2) attack 6 BID11 with a confidence parameter of 10. The CW2 attack tries to find the minimal change to the image, with respect to the L 2 norm, that will lead the image to be misclassified. It has been demonstrated to be one of the strongest extant adversarial attacks BID12.

Represent the short summary for finding the corresponding scientific article (long)

We present a method for interpreting black-box models by using instance-wise backward selection to identify minimal subsets of features that alone suffice to justify a particular decision made by the model.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose new objective measurement for evaluating explanations based on the notion of adversarial robustness. The evaluation criteria further allows us to derive new explanations which capture pertinent features qualitatively and quantitatively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, we tackle the problem of detecting samples that are not drawn from the training distribution, i.e., out-of-distribution (OOD) samples, in classification. Many previous studies have attempted to solve this problem by regarding samples with low classification confidence as OOD examples using deep neural networks (DNNs). However, on difficult datasets or models with low classification ability, these methods incorrectly regard in-distribution samples close to the decision boundary as OOD samples. This problem arises because their approaches use only the features close to the output layer and disregard the uncertainty of the features. Therefore, we propose a method that extracts the uncertainties of features in each layer of DNNs using a reparameterization trick and combines them. In experiments, our method outperforms the existing methods by a large margin, achieving state-of-the-art detection performance on several datasets and classification models. For example, our method increases the AUROC score of prior work (83.8%) to 99.8% in DenseNet on the CIFAR-100 and Tiny-ImageNet datasets. Deep neural networks (DNNs) have achieved high performance in many classification tasks such as image classification , object detection , and speech recognition ). However, DNNs tend to make high confidence predictions even for samples that are not drawn from the training distribution, i.e., out-of-distribution (OOD) samples . Such errors can be harmful to medical diagnosis and automated driving. Because it is not generally possible to control the test data distribution in real-world applications, OOD samples are inevitably included in this distribution. Therefore, detecting OOD samples is important for ensuring the safety of an artificial intelligence system . There have been many previous studies (; ; ; ; ;) that have attempted to solve this problem by regarding samples that are difficult to classify or samples with low classification confidence as OOD examples using DNNs. Their approaches work well and they are computationally efficient. The limitation of these studies is that, when using difficult datasets or models with low classification ability, the confidence of inputs will be low, even if the inputs are in-distribution samples. Therefore, these methods incorrectly regard such in-distribution samples as OOD samples, which in their poor detection performance , as shown in Figure 1. One cause of the abovementioned problem is that their approaches use only the features close to the output layer and the features are strongly related to the classification accuracy. Therefore, we use not only the features close to the output layer but also the features close to the input layer. We hypothesize that the uncertainties of the features close to the input layer are the uncertainties of the feature extraction and are effective for detecting OOD samples. For example, when using convolutional neural networks (CNNs), the filters of the convolutional layer close to the input layer extract features such as edges that are useful for in-distribution classification. In other words, indistribution samples possess more features that convolutional filters react to than OOD samples. Therefore, the uncertainties of the features will be larger when the inputs are in-distribution samples. Another cause of the abovementioned problem is that their approaches disregard the uncertainty of the features close to the output layer. We hypothesize that the uncertainties of the latent features close Baseline UFEL (ours) max softmax probability Baseline UFEL (ours) degree of uncertainty Figure 1: Comparison of existing and proposed methods. We visualized scatter plots of the outputs of the penultimate layer of a CNN that can estimate the uncertainties of latent features using the SVHN dataset . We used only classes 0, 1, and 2 for the training data. Classes 0, 1, 2, and OOD, indicated by red, yellow, blue, and black, respectively, were used for the validation data. We plot the contour of the maximum output of the softmax layer of the model. Left: Because the image of "204" includes the digits "2" and "0," the maximum value of the softmax output decreases because the model does not know to which class the image belongs. Right: The sizes of points in the scatter plots indicate the value of the combined uncertainties of features. We can classify the image of "204" as an in-distribution image according to the value of the combined uncertainties. to the output layer are the uncertainties of classification and are also effective for detecting OOD samples. For example, in-distribution samples are embedded in the feature space close to the output layer to classify samples. In contrast, OOD samples have no fixed regions for embedding. Therefore, the uncertainties of the features of OOD samples will be larger than those of in-distribution samples. Based on the hypotheses, we propose a method that extracts the Uncertainties of Features in Each Layer (UFEL) and combines them for detecting OOD samples. Each uncertainty is easily estimated after training the discriminative model by computing the mean and the variance of their features using a reparameterization trick such as the variational autoencoder and variational information bottleneck (; . Our proposal is agnostic to the model architecture and can be easily combined with any regular architecture with minimum modifications. We visualize the maximum values of output probability and the combined uncertainties of the latent features in the feature space of the penultimate layer in Figure 1 . The combined uncertainties of the features discriminate the in-distribution and OOD images that are difficult to classify. For example, although the images that are surrounded by the red line are in-distribution samples, they have low maximum softmax probabilities and could be regarded as OOD samples in prior work. Meanwhile, their uncertainties are smaller than those of OOD samples and they are regarded as in-distribution samples in our method. In experiments, we validate the hypothesis demonstrating that each uncertainty is effective for detecting OOD examples. We also demonstrate that UFEL can obtain state-of-the-art performance in several datasets including CIFAR-100, which is difficult to classify, and models including LeNet5 with low classification ability. Moreover, UFEL is robust to hyperparameters such as the number of in-distribution classes and the validation dataset. Methods based on the classification confidence proposed the baseline method to detect OOD samples without the need to further re-train and change the structure of the model. They define low-maximum softmax probabilities as indicating the low confidence of in-distribution examples and detect OOD samples using the softmax outputs of a pre-trained deep classifier. Building on this work, many models have recently been proposed. proposed ODIN, a calibration technique that uses temperature scaling in the Figure 2: Network structure of UFEL when using DenseNet. Black arrow: Extracting the variance of latent features using the reparameterization trick. Blue arrow: Combining these features. softmax function and adds small controlled perturbations to the inputs to widen the gap between indistribution and OOD features, which improves the performance of the baseline method.;;; also extended the baseline method. , we use the feature of maximum softmax probability as one of our features. Methods based on the uncertainty attempted to solve the problem of classifying in-distribution samples close to the decision boundary as OOD samples by distinguishing between data uncertainty and distributional uncertainty. Data uncertainty, or aleatoric uncertainty , is irreducible uncertainty such as class overlap, whereas distributional uncertainty arises because of the mismatch between training and testing distributions. They argue that the value of distributional uncertainty depends on the difference in the Dirichlet distribution of the categorical parameter. Further, they estimate the parameter of the Dirichlet distribution using a DNN and train the model with in-distribution and OOD datasets. The motivation for our work is similar to that of. In our work, the distribution of the logit of the categorical parameters is modeled as a Gaussian distribution, which enables us to train the model without an OOD dataset. Furthermore, we estimate the parameters of the Gaussian distribution of latent features close to the input layer. In this section, we present UFEL, which extracts the uncertainties of features in each layer and combines them for detecting OOD samples. First, we use the maximum of the softmax output, as in , as one of our features. Second, we also use the distribution of the categorical parameter, as in , using the uncertainty of logits. Furthermore, we use the uncertainty of the feature extraction extracted from the latent space close to the input layer because they will not be relevant to the classification accuracy. We probabilistically model the values of these features, estimate their uncertainties, and combine them. Let x ∈ X be an input, y ∈ Y = {1, · · ·, K} be its label, and l ∈ {1, · · ·, L} be the index of the block layers. The objective function of normal deep classification is as follows: where p(x, y) is the empirical data distribution, L is a cross entropy loss function, and f φ is a DNN. We use the following notation Figure 2. To extract the uncertainties of features in each layer, we model the lth block layer's output z l as a Gaussian whose parameters depend on the l-1th block layer's output z l−1 as follows:, where f φ l is the lth block layer, which outputs both the mean µ and covariance matrix Σ. In this paper, we use a diagonal covariance matrix to reduce the model parameters. We use the reparameterization trick to write z l = µ l + σ l, where, and is the Gaussian noise. Then, our objective function is as follows: where z 0 = x. Because of the reparameterization trick, the loss gradient is backpropagated directly through our model, and we can train our model like the regular classification models in Equation 1. Next, we explain the two methods of combining the features extracted in each layer. In the first method, we sum the uncertainties of each value of the features in each layer and linearly combine them. Because the feature maps of a convolutional block layer are three dimensional, each element is computed as z Moreover, because the output of a fully connected layer is one dimensional, each element is formed as z We use a weighted summation of the scale of each feature and the maximum value of the softmax scores as a final feature d LR as follows: We choose the parameter λ l by training a logistic regression (LR) using in-distribution and OOD validation samples. In the second method, we combine the features directly and nonlinearly using a CNN as follows: We train the CNN parameter θ with in-distribution and OOD validation samples using binary crossentropy loss. The detailed structures of the CNN are given in Table A.3. We use the values of these feature d(x) to test the performance of detecting OOD samples. In this section, we present the details of the experiments, which includes the datasets, metrics, comparison methods, and models. Because of space limitations, more details are given in Appendix A. Datasets We used several standard datasets for detecting OOD samples and classifying indistribution samples. The SVHN, CIFAR-10, and CIFAR-100 datasets were used as in-distribution datasets, whereas Tiny ImageNet (TIM), LSUN, iSUN, Gaussian noise, and uniform noise were used as OOD datasets. These data were also used in;. We applied standard augmentation (cropping and flipping) in all experiments. We used 5,000 validation images split from each training dataset and chose the parameter that can obtain the best accuracy in the validation dataset. We also used 68,257 training images from the SVHN dataset and 45,000 training images from the CIFAR-10 and CIFAR-100 datasets. All the hyperparameters of ODIN and UFEL were tuned on a separate validation set, which consists of 100 OOD images from the test dataset and 1,000 images from the in-distribution validation set. We tuned the parameters of the CNN in Equation 4 using 50 validation training images taken from the 100 validation images. The best parameters were chosen by validating the performance using the rest of 50 validation images. Finally, we tested the models with a test dataset that consisted of 10,000 in-distribution images and 9,900 OOD images. Evaluation metrics We used several standard metrics for testing the detection of OOD samples and the classification of in-distribution samples. We used TNR at 95% TPR, AUROC, AUPR, and accuracy (ACC), which were also used in Lee et al. (2017; . Comparison method We compare UFEL with the baseline and ODIN methods. For the baseline method, we used max k p(y = k|x) as the detection metric. For ODIN, we used the same detection metric and calibrated it using temperature scaling and small perturbations to the input. The temperature parameter T ∈ {1, 10, 100, 1000} and the perturbation parameter ∈ {0, 0.001, 0.005, 0.01, 0.05, 0.1} were chosen using the in-distribution and OOD validation datasets. Model training details We adopted LeNet5 and two state-of-the-art models, WideResNet and DenseNet , in this experiment. In all experiments, we used the same model and conditions to compare UFEL with existing methods. Only the structure used to extract the variance parameters differs. For LeNet5, we increased the number of channels of the original LeNet5 to improve accuracy. See Table A.3 for model details. We inserted the reparameterization trick to the second convolutional layer and the softmax layer. LeNet5 was trained using the Adam optimizer for 10 epochs and the learning rate was set to 5e-4. Both DenseNet and WideResNet were trained using stochastic gradient descent, with a Nesterov momentum of 0.9. We inserted the reparameterization trick to the first convolutional block, the second convolutional block, and the softmax layer. For WideResNet, we used a WideResNet with a depth of 40 and width of 4 (WRN-40-4), which was trained for 50 epochs. The learning rate was initialized to 0.1 and reduced by a factor of 10× after the 40th epoch. For DenseNet, we used a DenseNet with depth L = 100 (Dense-BC), growth rate of 12, and drop rate of 0. DenseNet-BC was trained for 200 epochs with batches of 64 images, and a weight decay of 1e-4 for the CIFAR-10 and CIFAR-100 datasets. It was trained for 10 epochs for the SVHN dataset. The learning rate was initialized to 0.1 and reduced by a factor of 10× after the 150th epoch. In this section, we demonstrate the performance of UFEL by conducting five experiments. In the first experiment, we show that UFEL performs better than the baseline and ODIN methods on several datasets and models. In the second experiment, we confirm that the features of UFEL have almost no relationship with the ACC. In the third experiment, we demonstrate that UFEL has a strong ability to detect OOD data, even if the number of classes of in-distribution data is small. In the fourth experiment, we confirm that UFEL is robust to the number of OOD samples, and in the fifth experiment, we test the performance of UFEL on unseen OOD datasets. The objective of these experiments is to show the uncertainties of the features obtained in each CNN layer distinguish the in-distribution and OOD data. Moreover, we obtain state-of-the-art performance for OOD sample detection by combining these features. Detecting OOD samples on several datasets and models In this experiment, we evaluate the performance of OOD detection using Equation 3 and Equation 4. In this study, var l is used to denote σ, and UFEL (CNN) denotes d CN N in Equation 4. We measured the detection performance using a DenseNet trained on CIFAR-100 when the iSUN dataset is used to provide the OOD images. Table 1 shows that var 1 and var 3 are strong features that, by themselves, can outperform ODIN. This indicates that the uncertainties of the feature extraction and classification are effective for detecting OOD samples. Moreover, the combination of these features yields state-of-the-art performance. In Table 2, we demonstrate that UFEL outperforms the baseline and ODIN methods on several datasets and models. Furthermore, UFEL is also slightly superior to them with respect to indistribution accuracy, which indicates that our model is robust to noise because of the reparameterization trick. Here, we do not report ODIN accuracy because the model of ODIN is the same as that of the baseline. We conducted this experiment three times and used the average of the . We used the CIFAR-10, CIFAR-100, and SVHN datasets as the in-distribution datasets and the other datasets as the OOD samples. Note that our UFEL outperformed the baseline and ODIN methods by a large margin, especially when using CIFAR-100, which is difficult to classify, or LeNet5 which Relationship between the performance of detecting OOD samples and in-distribution accuracy In this experiment, we show that the features of our method are not related to the in-distribution accuracy. We used CIFAR-10 as the in-distribution dataset and TIM as the OOD dataset. We trained DenseNet-BC for nine epochs and tested the performance at each epoch. As shown in Figure 3, each variance (var l) is less related to the accuracy than the baseline and ODIN methods. The var 1 of the feature close to the input layer has the highest ability to detect OOD samples in this experiment. Out-of-distribution: iSUN Figure 4: Plot of AUROC (y-axis) when changing the number of in-distribution dataset classes (xaxis). We used SVHN as in-distribution dataset, TIM, LSUN, and iSUN as OOD datasets, and the LeNet5 model. All plots were averaged over three runs and the error bar indicates one standard deviation. These also indicate that we can discriminate in-distribution and OOD examples when using a dataset that is difficult to classify. Detecting OOD samples while changing the number of in-distribution classes In this experiment, we show that UFEL is robust to the number of class labels. We used SVHN as indistribution dataset and changed the number of in-distribution classes in training as {0,1}, {0,1,2},..., {0,1,2,...,9}. We also used TIM, LSUN and iSUN datasets as OOD samples, and LeNet5 as a model. We compared the proposed method with the baseline and ODIN methods, as shown in Figure 4. This graph shows the AUROC score of each model when changing the number of training data classes. As this graph shows, UFEL outperforms other methods in all cases and is robust to the number of in-distribution classes, whereas the performance of ODIN drops as the number of class labels decreases. These suggest that UFEL is effective for small datasets because the number of samples can be decreased to one fifth of the original number when there are two in-distribution classes and the cost of label annotation is reduced. Detecting OOD samples while changing the number of OOD samples In this experiment, we present the performance of UFEL while changing the number of OOD validation examples. All the hyperparameters of ODIN and UFEL were tuned on a separate validation set, which consists of 30, 50, and 100 OOD images in the test dataset and 1,000 images from the in-distribution validation set. As shown in Figure 5, although UFEL (CNN) outperforms other methods including UFEL (LR) in most cases, it performs worse than ODIN in part of the because some tuning for OOD samples is needed. Meanwhile, UFEL (LR) outperforms prior methods constantly because the number of hyperparameters is small and tuning samples are almost unneeded. Figure 5: Plot of AUROC (y-axis) when changing the OOD dataset (x-axis). We used CIFAR-10 and CIFAR-100 as the in-distribution dataset. All plots are averaged over three runs and the error bar indicates one standard deviation. Generalization to unseen OOD dataset Because OOD validation samples might not be available in practice, we used only uniform noise as the validation OOD dataset and tested the ability of our model to detect another OOD dataset. We added a binary classification as a comparison method. This method was trained using an in-distribution dataset (positive) and uniform noise (negative). Table 3 shows that UFEL outperforms prior work in all cases and generalize well. Table 3 also indicates that the binary classification method does not generalize well because it cannot distinguish in-distribution dataset and OOD datasets TIM, LSUN, and iSUN, although it can distinguish Gaussian noise, which is similar to uniform noise. In this paper, we demonstrated that the uncertainties of features extracted in each hidden layer are important for detecting OOD samples. We combined these uncertainties to obtain state-of-the-art OOD detection performance on several models and datasets. The approach proposed in this paper has the potential to increase the safety of many classification systems by improving their ability to detect OOD samples. In future work, our model could be used in an unsupervised model by training it to minimize reconstruction error, which would avoid the need to use in-distribution labels to detect OOD samples. Furthermore, although we compared our model with ODIN, UFEL will perform better if we combine UFEL with ODIN because they are orthogonal methods. CIFAR. The CIFAR dataset contains 32 × 32 natural color images. The training set has 50,000 images and the test set has 10,000 images. CIFAR-10 has 10 classes, whilst CIFAR-100 has 100 classes. SVHN. The Street View Housing Numbers (SVHN) dataset contains 32 × 32 color images of house numbers. The training set has 604,388 images and the test set has 26,032 images. SVHN has 10 classes comprising the digits 0-9. TIM. The Tiny ImageNet dataset consists of a subset of ImageNet images . It contains 10,000 test images from 200 different classes. We downsampled the images to 32 × 32 pixels. LSUN. The Large-scale Scene UNderstanding (LSUN) dataset has 10,000 test images of 10 different scenes. We downsampled the images to 32 × 32 pixels. iSUN. The iSUN dataset consists of a subset of 8,925 SUN images. We downsampled the images to 32 × 32 pixels. Gaussian Noise. The Gaussian noise dataset consists of 10,000 random two-dimensional Gaussian noise images, where each value of every pixel is sampled from an i.i.d Gaussian distribution with mean 0.5 and unit variance. Uniform Noise. The uniform noise dataset consists of 10,000 images, where each value of every pixel is independently sampled from a uniform distribution on. In channels Out channels Ksize Stride Padding Conv2d 3 64 5 1 0 ReLU -----MaxPool2d ---2 -Conv2d 64 128 5 1 0 ReLU -----MaxPool2d ---2 -Linear 128*5*5 120 ---ReLU -----Linear 120 84 ---ReLU -----Linear 84 10 ---softmax -----

Represent the TLDR for finding the corresponding scientific article

We propose a method that extracts the uncertainties of features in each layer of DNNs and combines them for detecting OOD samples when solving classification tasks.

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.