Question: Represent the article for finding a single-sentence TLDR.

Question: Model interpretability and systematic, targeted model adaptation present central challenges in deep learning. In the domain of intuitive physics, we study the task of visually predicting stability of block towers with the goal of understanding and influencing the model's reasoning. Our contributions are two-fold. Firstly, we introduce neural stethoscopes as a framework for quantifying the degree of importance of specific factors of influence in deep networks as well as for actively promoting and suppressing information as appropriate. In doing so, we unify concepts from multitask learning as well as training with auxiliary and adversarial losses. Secondly, we deploy the stethoscope framework to provide an in-depth analysis of a state-of-the-art deep neural network for stability prediction, specifically examining its physical reasoning. We show that the baseline model is susceptible to being misled by incorrect visual cues. This leads to a performance breakdown to the level of random guessing when training on scenarios where visual cues are inversely correlated with stability. Using stethoscopes to promote meaningful feature extraction increases performance from 51% to 90% prediction accuracy. Conversely, training on an easy dataset where visual cues are positively correlated with stability, the baseline model learns a bias leading to poor performance on a harder dataset. Using an adversarial stethoscope, the network is successfully de-biased, leading to a performance increase from 66% to 88%. Intuitive physics in the deep learning community describes physical understanding acquired by neural networks in a data-driven as opposed to a rule-based manner: With an increasing amount of training examples, we expect an algorithm to develop a better understanding of its (physical) environment, especially when the task it is trained on is inherently linked to the physical rules governing the scene. However, what type of understanding the network develops highly depends on the types of scenarios it is confronted with and the task it is trying to solve. Furthermore, it depends on the network architecture, on regularisation techniques, on the training procedure, etc. As a , in contrast to a rule-based approach, it is often hard to assess what form of physical understanding a neural network has developed. We are specifically interested in whether the network uses visual cues as shortcuts which reflect correlations in the dataset but are incommensurate with the underlying laws of physics the network was intended to learn. In this paper, we specifically focus on stability prediction of block towers, a task which has gained interest in both the deep learning BID10 BID22 BID8 and the robotics community in recent years BID11 b). Images of towers of blocks stacked on top of each other are shown to a neural network. Its task is to predict whether the tower will fall over or not ing in a binary classification problem. End-to-end learning approaches as well as simulation-based approaches achieve super-human performance on a real dataset BID10 BID22 BID8. However, with investigation of trained deep learning models limited to occlusion-based attention analyses BID10 BID8, it is not clear to what extent neural networks trained on this task take into account physical principles such as centre-of-mass or whether they follow visual cues instead. To this end, we introduce a variation of the ShapeStacks dataset presented by BID8 which facilitates the analysis of the effects of visual cues on the learning process. The stethoscope framework. The main network (blue), comprised of an encoder and a decoder, is trained for global stability prediction of block towers. The stethoscope (orange), a two layered perceptron, is trained to predict a nuisance parameter (local stability) where the input is Z, a learned feature from an arbitrary layer of the main network. The stethoscope loss is back-propagated with weighting factor λ to the main network. The value of λ determines whether the stethoscope operates in analytic (λ " 0), auxiliary (λ ą 0) or adversarial manner (λ ă 0).Motivated by the need for an effective tool to understand and guide the physical reasoning of the neural network and inspired by prior research in interpretability, multi-task learning and adversarial training, we present neural stethoscopes as a unified framework for the interrogation and perturbation of task-specific information at any layer. A stethoscope can be deployed in a purely analytic fashion whereby a question is posed via a stethoscope loss which is not propagated back into the main network. It can also be used to promote or suppress specific information by deploying either an auxiliary or an adversarial training mechanism. The concept is illustrated in FIG0. We demonstrate that deploying an auxiliary stethoscope can be used to promote information conducive to the main task improving overall network performance. Conversely, we show that an adversarial stethoscope can mitigate a specific bias by effectively suppressing information. Moreover, the main network does not need to be changed in order to apply a neural stethoscope. In this work, we present two contributions: An in-depth analysis of the state-of-the-art approach for intuitive stability prediction. To that end, we also introduce an extension to the existing ShapeStacks dataset which will be made publicly available. A framework for interpreting, suppressing or promoting extraction of features specific to a secondary task unifying existing approaches from interpretability, auxiliary and adversarial learning. While we frame this work in the context of intuitive physics, questions regarding model interpretability and, consequently, systematic, targeted model adaptation find applicability in all domains of deep learning. For a study of two MNIST toy problems with neural stethoscopes, please see Appendix C. This work touches on two fields within deep learning: intuitive physics, specifically stability prediction, and targeted model adaptation/interpretation. Vision-based stability prediction of block towers has become a popular task for teaching and showcasing physical understanding of algorithms. Approaches include end-to-end deep learning algorithms BID10 BID8 as well as pipelines using computer vision to create an abstract state representation which can then be used by a physics simulator BID5 BID22. BID5 achieve impressive with a pipeline approach to robotic stone stacking. Interestingly, on a publicly available real-world dataset of block tower images BID10, the state-of-the-art is shared between an end-to-end learning approach BID8 ) and a pipeline method using a physics simulator BID22. While being much easier to implement, end-to-end approaches have the downside of being significantly harder to interpret. Interpretability brings at least two advantages: Trust by understanding the model's reasoning and therefore also its potential failure cases. Identification of potentials to improve the model. Both BID10 and BID8 conduct occlusion-based attention analyses. BID8 find that the focus of the algorithm's attention lies within a bounding box around the stability violation in 80% of the cases. While encouraging, which can be drawn regarding the network's understanding of physical principles are limited. Moreover, BID16 shows that attention analyses can be misleading: The Grad-CAM visualisation does not change even for artificially crafted adversarial examples which maximally confuse the classifier. Neural Stethoscopes The notion of passively interrogating and actively influencing feature representations in hidden layers of neural networks connects disparate fields including interpretability, auxiliary losses and multitask learning as well as adversarial training. We note that much of the required machinery for neural stethoscopes already exists in a number of sub-domains of deep learning. Work on auxiliary objectives BID9 as well as multi-task learning (e.g. BID3 BID4) commonly utilises dedicated modules with losses targeted towards a variety of tasks in order to optimise a representation shared across tasks. Based on this notion both deep supervision BID21 and linear classifier probes BID1 reinforce the original loss signal at various levels throughout a network stack. Although their work is restricted to reinforcing the learning signal via the same loss applied to the global network BID1 in particular demonstrate that the accessibility of information at a given layer can be determined -and promotedby formulating a loss applied locally to that layer. Conversely, in order to encourage representations invariant to components of the input signal, regularisation techniques are commonly utilised (e.g. BID17). To obtain invariance with respect to known systematic changes between training and deployment data, BID6; BID23 propose methods for domain adaptation. In order to prevent a model fitting to a specific nuisance factor, BID13 minimise the Maximum Mean Discrepancy between conditional distributions with different values of a binary nuisance variable in the conditioning set. This method is limited to discrete nuisance variables and its computational cost scales exponentially with the number of states. BID24 address both issues via adversarial training, optimising an encoding to confuse an additional discriminator, which aims to determine the values of nuisance parameters. This approach assumes that the nuisance variable is known and is an input to the main model during training and deployment. BID14 follow a similar approach, applying the discriminator to the output of the main model instead of its intermediate representations. We use stethoscopes to analyse and influence the learning process on the task of stability prediction, but present it in the following as a general framework which can be applied to any set of tasks. In supervised deep learning, we typically look for a function f θ: X Ñ Y with parameters θ that maps an input x P X to its target y P Y. Often the function internally computes one or more intermediate representations z P Z of the data. In this case, we rewrite f θ as the composition of the encoder h enc θ: X Ñ Z, which maps the input to the corresponding features z P Z, and the decoder h dec θ: Z Ñ Y, which maps features to the output. In this work, we consider only classification tasks so that Y is a finite set of labels (or a simplex in the case of a probabilistic output), but our approach generalises directly also to regression tasks. The information present in the input x might support multiple different tasks. By introducing a supplementary task into our training framework we hope to improve the performance of our network at the primary task. However, the impact of the supplementary task on the primary task is often difficult to determine, and in some cases can even detriment performance on the primary task. To this end, we propose neural stethoscopes, which allow us to interrogate the relationships between primary and supplementary tasks. In addition, the framework provides a tool to improve the network's performance at the primary task in light of both beneficial and detrimental supplementary tasks through the introduction of auxiliary and adversarial losses respectively promoting or suppressing extraction of features related to the supplemental task by the encoder h is trained on a supplemental task with targets s P S and not for the main objective. We define two loss functions: L y pθq, which measures the discrepancy between predictions f θ and the true task y and L s pθ, ψq, which measures the performance on the supplemental task. The weights of the stethoscope are updated as´∆ψ 9 ∇ ψ L s pθ, ψq to minimise L s pθ, ψq and the weights of the main network as´∆θ 9 ∇ θ L y,s pθ, ψq to minimise the energy DISPLAYFORM0 By choosing different values for the constant λ we obtain three very different use cases:Analytic Stethoscope (λ " 0) Here, the gradients of the stethoscope, which acts as a passive observer, are not used to alter the main model. This setup can be used to interrogate learned feature representations: if the stethoscope predictions are accurate, the features can be used to solve the task. Auxiliary Stethoscope (λ ą 0) The encoder is trained with respect to the stethoscope objective, hence enforcing correlation between main network and supplemental task. This setup is related to learning with auxiliary tasks, and helpful if we expect the two tasks to be beneficially related. Adversarial Stethoscope (λ ă 0) By setting λ ă 0, we train the encoder to maximise the stethoscope loss (which the stethoscope still tries to minimise), thus encouraging independence between main network and supplemental tasks. This is effectively an adversarial training framework and is useful if features required to solve the stethoscope task are a detrimental nuisance factor. For the analytic stethoscope, to fairly compare the accessibility of information with respect to a certain task in different feature representations, we set two criteria: FORMULA0 The capacity of the stethoscope architecture has to be constant regardless of the dimensions of its input. FORMULA1 The stethoscope has to be able to access each neuron of the input separately. We guarantee this by fully connecting the input with the first layer of the stethoscope using a sparse matrix. This matrix has a constant number of non-zero entries (criterion 1) and connects every input unit as well as every output unit at least once (criterion 2). For a more detailed description, see Appendix A.In auxiliary and adversarial mode, we attach the stethoscope to the main network's last layer before the logits in a fully connected manner. This setup proved to have the highest impact on the learning process of the main network. The stethoscope itself is implemented as a two-layer perceptron with ReLU activation and trained with sigmoid or softmax cross-entropy loss on its task S.For numerical stability, the loss of the encoder in the adversarial setting is rewritten as DISPLAYFORM1 where Lspθ, ψq is the stethoscope loss with flipped labels. The objective is similar to the confusion loss formulation utilised in GANs to avoid vanishing gradients when the discriminator's performance is high BID7. Previous work has shown that neural networks are highly capable of learning physical tasks such as stability prediction. However, unlike approaches using physics simulators BID5 BID22, with pure-learning based approaches, it is hard to assess what reasoning they follow and whether they gain a sound understanding of the physical principles or whether they learn to take short-cuts following visual cues based on correlations in the training data. Occlusion-based attention analyses are a first step in this direction, but the insights gained from this are limited BID10 BID8. In this section, we follow the state-of-art approach on visual stability prediction of block towers and examine as well as influence its learning behaviour. We introduce a variation of the ShapeStacks dataset from BID8 which is particularly suited to study the dependence of network predictions on visual cues. We then examine how suppressing or promoting the extraction of certain features influences the performance of the network using neural stethoscopes. Dataset As shown in BID8, a single-stranded tower of blocks is stable if, and only if, at every interface between two blocks the centre of mass of the entire tower above is supported by the convex hull of the contact area. If a tower satisfies this criterion, i.e., it does not collapse, we call it globally stable. To be able to quantitatively assess how much the algorithm follows visual cues, we introduce a second label: We call a tower locally stable if, and only if, at every interface between two blocks, the centre of mass of the block immediately above is supported by the convex hull of the contact area. Intuitively, this measure describes, if taken on its own without any blocks above, each block would be stable. We associate binary prediction tasks y G and y L to respective global and local stability where label y " 0 indicates stability and y " 1 instability. Global and local instability are neither mutually necessary nor sufficient, but can easily be confused visually which is demonstrated by our experimental . Based on the two factors of local and global stability, we create a simulated dataset 1 with 4,000 block tower scenarios divided into four qualitative categories (cf. FIG2 . The dataset is divided into an easy subset, where local and global stability are always positively correlated, and a hard subset, where this correlation is always negative. The dataset will be made available online. Model We choose the Inception-v4 network BID18 as it yields state-of-the-art performance on stability prediction BID8 . The model is trained in a supervised setting using example tuples px, y G, y L q consisting of an image x and its global and local stability labels y G and y L . Classification losses use sigmoid cross entropy. We use the RMSProp optimiser BID19) throughout all our experiments. Local Stability as a Visual Cue Based on the four categories of scenarios described in FIG2, we conduct an initial set of experiments to gauge the influence of local stability on the network predictions. If local stability had, as it would be physically correct, no influence on the network's prediction of global stability, the performance of the network should be equal for easy and hard scenarios, regardless of the training split. However, FIG3 shows a strong influence of local stability on the prediction performance. When trained on the entire, balanced data set, the error rate is three times higher for hard than for easy scenarios (6% vs. 2%). When trained on easy scenarios only, the error rate even differs by a factor of 13. Trained on hard scenarios only, the average performance across all four categories is on the level of random chance (51%), indicating that negatively correlated local and global stability imposes a much harder challenge on the network. After demonstrating the influence of local stability on the task of global stability prediction we turn our attention to the use of neural stethoscopes to quantify and actively mitigate this influence.5a 5b 5c 5d 6a 6b 6c 6d 6e 6f 6g 6h 7a 7b 7c 7d layer 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 accuracy local stability origin of global instability global stability Figure 4: Analysis of the prediction performance throughout the Inception-v4 network trained on predicting global stability. We report average performances on balanced test data after 50 epochs of stethoscope training. All stethoscopes have been attached to the respective activation tensors 3 with sparse connection matrices as described in Section 3.Task Relationship Analysis We seek to quantify the correlation of features extracted for global stability prediction with the task of local instability detection. We train the main network on global stability prediction while attaching stethoscopes to multiple layers. The stethoscope is trained on three tasks: global stability prediction (binary), local stability (binary) and origin of global instability, particularly the interface at which this occurs (n-way one-hot). Figure 4 reveals that the network gradually builds up features which are conducive to both global and local stability prediction. However, the performance of local stability prediction peaks at the activation tensor after layer 7a whereas the performance for global stability prediction (both binary and n-way) keeps improving. This is in line with the governing physical principles of the scenarios and yields the conjecture that the information about local stability can be considered a nuisance factor whereas information about the global site of stability violation can serve as a complementary factor for the main task. We now test the hypothesis that fine-grained labels of instability locations help the main network to grasp the correct physical concepts. To that end, The main network is trained on binary global stability labels while the stethoscope is trained on more fine grained labels -origin of global stability (n-way). The network was trained on hard scenarios only but evaluated on all. The dashed lines represent baselines for λ " 0.we consider the setup from FIG3 where the training data only consists of hard scenarios with a baseline performance of 51%. The main network is trained on global stability while the stethoscope is trained on predicting the origin of global instability, namely the interface at which the instability occurs. FIG4 shows that auxiliary training substantially improves the performance for weighting parameters λ P r0.5, 16s. However, for very small values of λ, the contribution of the additional loss term is too small while for large values, performance deteriorates to the level of random chance as a of the primary task being far out-weighted by the auxiliary task. FIG3 indicate that the network might use local stability as a visual cue to make biased assumptions about global stability. We now investigate whether it is possible to debias the network by forcing it to pay less attention to local stability. To that end, we focus on the scenario shown in FIG3, where we only train the network on global stability labels for easy scenarios. As shown in FIG3, the performance of the network suffers significantly when tested on hard scenarios where local and global stability labels are inversely correlated. The hypothesis is that forcing the network not to focus on local stability weakens this bias. In FIG6, we use active stethoscopes (λ ‰ 0) to test this hypothesis. We train a stethoscope on local stability on labels of all categories (in a hypothetical scenario where local labels are easier to obtain than global labels) and use both the adversarial and the auxiliary setup in order to test the influence of suppressing and promoting accessibility of information relevant for local stability in the encoded representation, respectively. In FIG6, the of both adversarial and auxiliary training are compared to the baseline of λ " 0, which is equivalent to the analytic stethoscope setup. FIG6 shows that adversarial training does indeed partly remove the bias and significantly increases the performance of the main network on hard scenarios while maintaining its high accuracy on easy scenarios. The bias is removed more and more with increasing magnitude of the weighting factor λ up to a point where further increasing λ jeopardises the performance on the main task as the encoder puts more focus on confusing the stethoscope than on the main task (in our experiments this happens at λ « 10 1). Interestingly, the correlation of local stability and global stability prediction rises at this point as for pushing the stethoscope below 50% (random guessing), the main network has to extract information about local stability. Naturally, the performance of the stethoscope continuously decreases with increasing λ as the encoder puts more and more focus on confusing the stethoscope. This scenario could also be seen from the perspective of feeding additional information into the network, which could profit from more diverse training data. However, FIG6 shows that naively using an auxiliary setup to train the network on local stability worsens the bias. With increasing λ and increasing performance of the stethoscope, the network slightly improves its performance on easy scenarios while accuracy deteriorates on hard scenarios. These observations are readily explained: local and global stability are perfectly correlated in the easy scenarios, i.e., the (global) training data. The network is essentially free to chose whether to select local or global features when predicting global stability. Auxiliary training on local stability further shifts the focus to local features. When tested on hard scenarios, where local and global stability are inversely correlated, the network will therefore perform worse when it has learned to rely on local features. We study the state-of-the-art approach for stability prediction of block towers and test its physical understanding. To that end, we create a new dataset and introduce the framework of neural stethoscopes unifying multiple threads of work in machine learning related to analytic, auxiliary and adversarial probing of neural networks. The analytic application of stethoscopes allows measuring relationships between different prediction tasks. We show that the network trained on stability prediction also obtains a more fine-grained physical understanding of the scene (origin of instability) but at the same time is susceptible to potentially misleading visual cues (i.e., local stability). In addition to the analysis, the auxiliary and adversarial modes of the stethoscopes are used to support beneficial complementary information (origin of instability) and suppress harmful nuisance information (visual cues) without changing the network architecture of the main predictor. This yields substantial performance gains in unfavourable training conditions where data is biased or labels are partially unavailable. We encourage the use of neural stethoscopes for other application scenarios in the future as a general tool to analyse task relationships and suppress or promote extraction of specific features. This can be done by collecting additional labels or using existing multi-modal data. This is a detailed explanation of how to connect the first layer of the stethoscope with any layer of the main network as described in Section 3. The goal is to guarantee equal capacity of the stethoscope regardless of the dimensions of the layer it is connected to in order to obtain comparable . Let's call the flattened feature vector from the main network x. In the following we describe how to compute the sparse matrix M which reduces the full vector x to a smaller vector x 1 while adding biases: DISPLAYFORM0 M is a sparse matrix with a constant number of non-zero entries (n_non_zero) which are trainable. The hyperparameter n_non_zero, which determines the capacity of this additional linear layer, is chosen as an integer multiple of the length of x 1 and has to be greater or equal than the number of features in any of the layers of the main network which the stethoscope is attached to. The matrix M is generated in a way so that it fulfils the following two guarantees:• Each input is connected at least once. I.e., no column only contains zeroes.• Each output is connected an equal amount of times. I.e., all rows have the same number of non-zero entries. FIG6 shows that adversarial training increases performance given a biased training dataset. While hypothetically the network could learn to completely discard information about local stability when trying to confuse the adversarial stethoscope, it could also simply reduce its accessibility. In an additional experiment FIG7, where the main network (blue line) is fixed after adversarial training for 120 epochs (grey dashed line), the performance of the stethoscope on local stability (orange) is fully recovered to the level of the baseline for a purely analytic stethoscope (dashed orange line). We therefore argue that the information is not removed, but instead made less accessible to the stethoscope by constantly shifting the representation as explained in BID15. However, the ing performance increase for the main task indicates that this behaviour also makes information related to local stability less accessible to the decoder of the main network. Hence, although the adversarial setup does not reach final, stable equilibria (cf. BID15), it decreases the dependence of the decoder on features particular to local stability. Hence, further improvements for stabilising GAN training could additionally improve performance in adversarial stethoscope use-cases BID15 BID7 BID2. This section introduces two intuitive examples and serves to illustrate the challenge with biased training datasets. We conduct two toy experiments built on the MNIST dataset to demonstrate the issue of varying correlation between nuisance parameters and ground truth labels in training and test datasets and to demonstrate the use of an adversarial stethoscope to suppress this information and mitigate overfitting. Both experiments build on the provision of additional hints as network input. In the first experiment, the hint is a separately handled input in the form of a one-hot vector whereas in the second experiment, the hint is directly incorporated in the pixel space of the MNIST images. In both experiments, the network is trained on the loss as defined in Equation and the gradients are used to update the weights of the different parts of the network (encoder(s), decoder, stethoscope) as defined in Section 3. Hence, λ ă 0 denotes adversarial training and λ ą 0 auxiliary training. Hint as Separate One-Hot Vector In the first example, the hint labels have the same format as the true labels of the input image and we run experiments with varying correlation between hint and true label in the training set. Given high correlation, the network can learn an identity transformation between hint and output and ignore the image. Now, at test time, the hint is completely independent of the true label and hence contains no useful information. The experiment mimics challenges with biased datasets as the supplemental task s P S (see Section 3) has labels y s which are correlated to the labels y of the main objective during training but not at test time (p train py s |yq ‰ p test py s |yq). Hence, an approximation of the correlations in the training data will not generalise to data during deployment. To that end, we introduce a parameter q h denoting the quality of the hints (i.e. their correlation with the true labels) during training time where for q h " 1.0, the hints are always correct (during training), for q h " 0.5, the hints are correct for 50% of the training examples and so on. Varying the hint quality enables us to investigate biases of increasing strength. In this setup FIG8 ), both input streams are separated via independent encoders. Let h enc1 θ and h enc2 θ respectively denote image and hint encoders. The stethoscope only accesses the hint encoding in this example, which simplifies its task and enables us to demonstrate the effect of active stethoscopes without affecting the image encoding. The hint encoder multiplies the one-hot hint vector with a scalar weight. Hence, a weight close to zero would effectively hide the hint information from the main network, explicitly making the task of the stethoscope h ψ S more difficult. The image encoder consists of two convolutional layers followed by two fully connected layers with 256 units where each layer is followed by leaky ReLU activations. The decoder h dec θ, which acts as a digit classifier, is comprised of a single fully connected layer and receives the image encoding concatenated with the encoded hint vector as input. Figure 9: Influence of adversarial training on classifier performance for toy experiment variant 1 described in FIG8. Each curve represents the for different hint qualities h q. The bold lines indicate the mean of 100 runs and the shaded areas mark the student-t 1 σ confidence intervals. In (a) the network is evaluated against the ground truth. In (b), the output of the network is compared to the hint labels during test time. Hence, a high accuracy in (b) shows strong overfitting while a high accuracy in (a) shows that the network learned the actual task of classifying digits from images. With adversarial training of the stethoscope, the encoder learns to suppress information about the hints forcing the classifier to purely rely on image data. As can be observed in Figure 9b, for perfect hints (q h " 1.0) and no adversarial training, the decoder is highly accurate in predicting the hint (instead of the actual label), suggesting that the classifier is effectively independent of the input image and purely relies on the hint. With adversarial training, however, we can indeed force the encoder to hide the hint, therefore forcing it to learn to classify digits from images. As expected, with increasing hint quality, we need to increase the weight of adversarial training so as to discourage the encoder from relying on the hint. FIG0: Influence of adversarial training on classifier performance for the second MNIST experiment. The adversarial training is less effective as both the stethoscope and the main network directly access a single encoding. High adversarial weights can strongly degrade performance on the main task as a trivial solution would be to suppress all information about the input. Each curve represents the for different hint qualities h q. The bold lines indicate the mean of 20 runs and the shaded areas mark the student-t 1 σ confidence intervals. In the second variant of the MNIST experiment, hints are provided at training time as a set of high intensity pixels in the input images as opposed to the explicit concatenation in the previous example. This better reflects the typical scenario where the goal is to disentangle the nuisance hints from the informative information related to the main task. Here, the main network is a two-layer multi-layer perceptron (MLP) with the stethoscope attached after the first hidden layer. The simpler architecture (compared to the one used in the first toy problem) was chosen in order to lower the performance on vanilla MNIST classification in order to see clearer effects. Note that in this setup, giving the same labels to the hints as for the main task in the training scenario makes the two tasks indistinguishable (in particular for perfect hint quality h q " 1.0). We therefore introduce a higher number of hint classes with 100 different hints, each of them related to a single digit in a many-to-one mapping, such that theoretical suppression of hint information is possible without degrading main performance. FIG0 shows the digit classification accuracy for the main network where the stethoscope is applied with weights ranging in both the positive and negative directions to reflect auxiliary and adversarial use of the stethoscope respectively. When using a positive stethoscope loss, which encourages h enc θ to focus more on the artificial hints over the actual digit features, the test accuracy degrades. This is expected for this toy example as the hints have zero correlation with the digit labels at test time. In the negative case, h enc θ serves as an adversary to the stethoscope effectively benefiting the main objectives by removing the misleading hint information leading to an improved test accuracy. However, as the magnitude of the adversarial strength increases the gains quickly erode as the encoder begins to suppress information relevant for the main task.

represent the tldr for finding the corresponding scientific article
For example, We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated. should be similar to We present a new approach to assessing the robustness of neural networks based on estimating the proportion of inputs for which a property is violated. Specifically, we estimate the probability of the event that the property is violated under an input model. Our approach critically varies from the formal verification framework in that when the property can be violated, it provides an informative notion of how robust the network is, rather than just the conventional assertion that the network is not verifiable. Furthermore, it provides an ability to scale to larger networks than formal verification approaches. Though the framework still provides a formal guarantee of satisfiability whenever it successfully finds one or more violations, these advantages do come at the cost of only providing a statistical estimate of unsatisfiability whenever no violation is found. Key to the practical success of our approach is an adaptation of multi-level splitting, a Monte Carlo approach for estimating the probability of rare events, to our statistical robustness framework. We demonstrate that our approach is able to emulate formal verification procedures on benchmark problems, while scaling to larger networks and providing reliable additional information in the form of accurate estimates of the violation probability. The robustness of deep neural networks must be guaranteed in mission-critical applications where their failure could have severe real-world implications. This motivates the study of neural network verification, in which one wishes to assert whether certain inputs in a given subdomain of the network might lead to important properties being violated BID29 BID1. For example, in a classification task, one might want to ensure that small perturbations of the inputs do not lead to incorrect class labels being predicted BID23 BID8.The classic approach to such verification has focused on answering the binary question of whether there exist any counterexamples that violate the property of interest. We argue that this approach has two major drawbacks. Firstly, it provides no notion of how robust a network is whenever a counterexample can be found. Secondly, it creates a computational problem whenever no counterexamples exist, as formally verifying this can be very costly and does not currently scale to the size of networks used in many applications. To give a demonstrative example, consider a neural network for classifying objects in the path of an autonomous vehicle. It will almost certainly be infeasible to train such a network that is perfectly robust to misclassification. Furthermore, because the network will most likely need to be of significant size to be effective, it is unlikely to be tractable to formally verify the network is perfectly robust, even if such a network exists. Despite this, it is still critically important to assess the robustness of the network, so that manufacturers can decide whether it is safe to deploy. To address the shortfalls of the classic approach, we develop a new measure of intrinsic robustness of neural networks based on the probability that a property is violated under an input distribution model. Our measure is based on two key insights. The first is that for many, if not most, applications, full formal verification is neither necessary nor realistically achievable, such that one actually desires a notion of how robust a network is to a set of inputs, not just a binary answer as to whether it is robust or not. The second is that most practical applications have some acceptable level of risk, such that it is sufficient to show that the probability of a violation is below a certain threshold, rather than confirm that this probability is exactly zero. By providing a probability of violation, our approach is able to address the needs of applications such as our autonomous vehicle example. If the network is not perfectly robust, it provides an explicit measure of exactly how robust the network is. If the network is perfectly robust, it is still able to tractability assert that a violation event is "probably-unsatisfiable". That is it is able to statistically conclude that the violation probability is below some tolerance threshold to true zero, even for large networks for which formal verification would not be possible. Calculating the probability of violation is still itself a computationally challenging task, corresponding to estimating the value of an intractable integral. In particular, in most cases, violations of the target property constitute (potentially extremely) rare events. Consequently, the simple approach of constructing a direct Monte Carlo estimate by sampling from the input model and evaluating the property will be expensive and only viable when the event is relatively common. To address this, we adapt an algorithm from the Monte Carlo literature, adaptive multi-level splitting (AMLS) BID9 BID18, to our network verification setting. AMLS is explicitly designed for prediction of rare events and our adaptation means that we are able to reliably estimate the probability of violation, even when the true value is extremely small. Our ing framework is easy to implement, scales linearly in the cost of the forward operation of the neural network, and is agnostic both to the network architecture and input model. Assumptions such as piecewise linearity, Lipschitz continuity, or a specific network form are not required. Furthermore, it produces a diversity of samples which violate the property as a side-product. To summarize, our main contributions are:• Reframing neural network verification as the estimation of the probability of a violation, thereby providing a more informative robustness metric for non-verifiable networks; • Adaptation of the AMLS method to our verification framework to allow the tractable estimation of our metric for large networks and rare events; • Validation of our approach on several models and datasets from the literature. The literature on neural network robustness follows two main threads. In the optimization community, researchers seek to formally prove that a property holds for a neural network by framing it as a satisfiability problem BID29 ), which we refer to as the classical approach to verification. Such methods have only been successfully scaled beyond one hidden layer networks for piecewise linear networks BID2 BID14, and even then these solutions do not scale to, for example, common image classification architectures with input dimensions in the hundreds, or apply to networks with nonlinear activation functions BID1. Other work has sought approximate solutions in the same general framework but still does not scale to larger networks BID19 BID28 BID12. As the problem is NP-hard BID14, it is unlikely that an algorithm exists with runtime scaling polynomially in the number of network nodes. In the deep learning community, research has focused on constructing and defending against adversarial attacks, and by estimating the robustness of networks to such attacks. BID26 recently constructed a measure for robustness to adversarial attacks estimating a lower bound on the minimum adversarial distortion, that is the smallest perturbation required to create an adversarial example. Though the approach scales to large networks, the estimate of the lower bound is often demonstratively incorrect: it is often higher than an upper bound on the minimum adversarial distortion BID7. Other drawbacks of the method are that it cannot be applied to networks that are not Lipschitz continuous, it requires an expensive gradient computation for each class per sample, does not produce adversarial examples, and cannot be applied to non-adversarial proper-ties. The minimum adversarial distortion is also itself a somewhat unsatisfying metric for many applications, as it conveys little information about the prevalence of adversarial examples. In other work spanning both communities BID5 BID25 BID27, researchers have relaxed the satisfiability problem of classical verification, and are able to produce certificates-of-robustness for some samples (but not all that are robust) by giving a lowerbound on the minimal adversarial distortion. Despite these methods scaling beyond formal verification, we note that this is still a binary measure of robustness with limited informativeness. An orthogonal track of research investigates the robustness of reinforcement learning agents to failure BID11 BID15. For instance, concurrent work to ours BID24 takes a continuation approach to efficiently estimating the probability that an agent fails when this may be a rare event. To help elucidate our problem setting, we consider the ACASXU dataset BID14 from the formal verification literature. A neural network is trained to predict one of five correct steering decisions, such as "hard left," "soft left," etc., for an unmanned aircraft to avoid collision with a second aircraft. The inputs x describe the positions, orientations, velocities, etc. of the two aircraft. Ten interpretable properties are specified along with corresponding constraints on the inputs, for which violations correspond to events causing collisions. Each of these properties is encoded in a function, s, such that it is violated when s(x) ≥ 0. The formal verification problem asks the question, "Does there exist an input x ∈ E ⊆ X in a constrained subset, E, of the domain such that the property is violated?" If there exists a counterexample violating the property, we say that the property is satisfiable (SAT), and otherwise, unsatisfiable (UNSAT).Another example is provided by adversarial properties from the deep learning literature on datasets such as MNIST. Consider a neural network f θ (x) = Softmax(z(x)) that classifies images, x, into C classes, where the output of f gives the probability of each class. Let δ be a small perturbation in an l p -ball of radius, that is, δ p <. Then x = x + δ is an adversarial example for x if arg max i z(x) i = arg max i z(x) i, i.e. the perturbation changes the prediction. Here, the property function is s(x) = max i =c (z(x) i − z(x) c ), where c = arg max j z(x) j and s(x) ≥ 0 indicates that x is an adversarial example. Our approach subsumes adversarial properties as a specific case. The framework for our robustness metric is very general, requiring only a) a neural network f θ, b) a property function s(x; f, φ), and c) an input model p(x). Together these define an integration problem, with the main practical challenge being the estimation of this integral. Consequently, the method can be used for any neural network. The only requirement is that we can evaluate the property function, which typically involves a forward pass of the neural network. The property function, s(x; f θ, φ), is a deterministic function of the input x, the trained network f θ, and problem specific parameters φ. For instance, in the MNIST example, φ = arg max i f θ (x) i is the true output of the unperturbed input. Informally, the property reflects how badly the network is performing with respect to a particular property. More precisely, the event DISPLAYFORM0 represents the property being violated. Predicting the occurrence of these, typically rare, events will be the focus of our work. We will omit the dependency on f θ and φ from here on for notional conciseness, noting that these are assumed to be fixed and known for verification problems. The input model, p(x), is a distribution over the subset of the input domain that we are considering for counterexamples. For instance, for the MNIST example we could use p(x; x) ∝ 1 (x − x p ≤) to consider uniform perturbations to the input around an l p -norm ball with radius. More generally, the input model can be used to place restrictions on the input domain and potentially also to reflect that certain violations might be more damaging than others. Together, the property function and input model specify the probability of failure through the integral DISPLAYFORM1 This integral forms our measure of robustness. The integral being equal to exactly zero corresponds to the classical notion of a formally verifiable network. Critically though, it also provides a measure for how robust a non-formally-verifiable network is. Our primary goal is to estimate in order to obtain a measure of robustness. Ideally, we also wish to generate example inputs which violate the property. Unfortunately, the event E is typically very rare in verification scenarios. Consequently, the estimating the integral directly using Monte Carlo, DISPLAYFORM0 is typically not feasible for real problems, requiring an impractically large number of samples to achieve a reasonable accuracy. Even when E is not a rare event, we desire to estimate the probability using as few forward passes of the neural network as possible to reduce computation. Furthermore, the dimensionality of x is typically large for practical problems, such that it is essential to employ a method that scales well in dimensionality. Consequently many of the methods commonly employed for such problems, such as the cross-entropy method BID22 BID3, are inappropriate due to relying on importance sampling, which is well known to scale poorly. As we will demonstrate empirically, a less well known but highly effective method from the statistics literature, adaptive multi-level splitting (AMLS) BID13 BID9, can be readily adapted to address all the aforementioned computational challenges. Specifically, AMLS is explicitly designed for estimating the probability of rare events and our adaptation is able to give highly accurate estimates even when the E is very rare. Furthermore, as will be explained later, AMLS also allows the use of MCMC transitions, meaning that our approach is able to scale effectively in the dimensionality of x. A further desirable property of AMLS is that it produces property-violating examples as a side product, namely, it produces samples from the distribution DISPLAYFORM1 Such samples could, in theory, be used to perform robust learning, in a similar spirit to BID8 and BID16. Multi-level splitting BID13 divides the problem of predicting the probability of a rare event into several simpler ones. Specifically, we construct a sequence of intermediate targets, DISPLAYFORM0, to bridge the gap between the input model p(x) and the target π(x). For any choice of the intermediate levels, we can now represent equation FORMULA1 through the following factorization, DISPLAYFORM1 Provided consecutive levels are sufficiently close, we will be able to reliably estimate each P k by making use of the samples from one level to initialize the estimation of the next. Our approach starts by first drawing N samples, {x DISPLAYFORM2, from π 0 (·) = p(·), noting that this can be done exactly because the perturbation model is known. These samples can then be used to estimate P 1 using simple Monte Carlo, DISPLAYFORM3 where DISPLAYFORM4 In other words, P 1 is the fraction of these samples whose property is greater than L 1. Critically, by ensuring the value of L 1 is sufficiently small for {s(x n) ≥ L 1 } to be a common event, we can ensureP 1 is a reliable estimate for moderate numbers of samples N. Adaptive multi-level splitting with termination criterion DISPLAYFORM0 Evaluate and sort {s(x DISPLAYFORM1 log(I) ← log(I) + log(P k) Updating integral estimate 10:if log(I) < log(Pmin) then return (∅, −∞) end if Final estimate will be less than log(Pmin) 11:Initialize {x DISPLAYFORM2 by resampling with replacement N times from {x DISPLAYFORM3 Apply M MH updates separately to each x (k) n using g(x |x) 13:[Optional] Adapt g(x |x) based on MH acceptance rates 14: end while 15: return ({x DISPLAYFORM4 To estimate the other P k, we need to be able to draw samples from π k−1 (·). For this we note that if {x DISPLAYFORM5 are distributed according to π k−2 (·), then the subset of these samples for which DISPLAYFORM6 Furthermore, setting L k−1 up to ensure this event is not rare means a significant proportion of the samples will satisfy this property. To avoid our set of samples shrinking from one level to the next, it is necessary to carry out a rejuvenation step to convert this smaller set of starting samples to a full set of size N for the next level. To do this, we first carry out a uniform resampling with replacement from the set of samples satisfying s(x (k−1) n ) ≥ L k to generate a new set of N samples which are distributed according to π k (·), but with a large number of duplicated samples. Starting with these samples, we then successively apply M Metropolis-Hastings (MH) transitions targeting π k (·) separately to each sample to produce a fresh new set of samples {x DISPLAYFORM7 (see Appendix A for full details). These samples can then in turn be used to form a Monte Carlo estimate for P k, DISPLAYFORM8 along with providing the initializations for the next level. Running more MH transitions decreases the correlations between the set of samples, improving the performance of the estimator. We have thus far omitted to discuss how the levels L k are set, other than asserting the need for the levels to be sufficiently close to allow reliable estimation of each P k. Presuming that we are also free to choose the number of levels K, there is inevitably a trade-off between ensuring that each {s(X) ≥ L k } is not rare given {s(X) ≥ L k−1 }, and keeping the number of levels small to reduce computational costs and avoid the build-up of errors. AMLS BID9 builds on the basic multi-level splitting process, providing an elegant way of controlling this trade-off by adaptively selecting the level to be the minimum of 0 and some quantile of the property under the current samples. The approach terminates when the level reaches zero, such that L K = 0 and K is a dynamic parameter chosen implicitly by the adaptive process. Choosing the ρth quantile of the values of the property in discarding a fraction (1 − ρ) of the chains at each step of the algorithm. This allows explicit control of the rarity of the events to keep them at a manageable level. We note that if all the sample property values are distinct, then this approach gives P k = ρ, ∀k < K. To give intuition to this, we can think about splitting up log(I) into chunks of size log(ρ). For any value of log(I), there is always a unique pair of values {K, P K} such that log(I) = K log(ρ) + log(P K), K ≥ 0 and P K < ρ. Therefore the problem of estimating I is equivalent to that of estimating K and P K. The application of AMLS to our verification problem presents a significant complicating factor in that the true probability of our rare event might be exactly zero. Whenever this is the case, the basic AMLS approach outlined in BID9 will never terminate as the quantile of the property will never rise above zero; the algorithm simply produces closer and closer intermediate levels as it waits for the event to occur. To deal with this, we introduce a termination criterion based on the observation that AMLS's running estimate for I monotonically decreases during running. Namely, we introduce a threshold probability, P min, below which the estimates will be treated as being numerically zero. We then terminate the algorithm if I < P min and return I = 0, safe in the knowledge that even if the algorithm would eventually generate a finite estimate for I, this estimate is guaranteed to be less than P min.Putting everything together, gives the complete method as shown in Algorithm 1. See Appendix B for low-level implementation details. In our first experiment 1, we aim to test whether our robustness estimation framework is able to effectively emulate formal verification approaches, while providing additional robustness information for SAT properties. In particular, we want to test whether it reliably identifies properties as being UNSAT, for which I = 0, or SAT, for which I > 0. We note that the method still provides a formal demonstration for SAT properties because having a non-zero estimate for I indicates that at least one counterexample has been found. Critically, it further provides a measure for how robust SAT properties are, through its estimate for I.We used the COLLISIONDETECTION dataset introduced in the formal verification literature by BID4. It consists of a neural network with six inputs that has been trained to classify two car trajectories as colliding or non-colliding. The architecture has 40 linear nodes in the first layer, followed by a layer of max pooling, a ReLU layer with 19 hidden units, and an output layer with 2 hidden units. Along with the dataset, 500 properties are specified for verification, of which 172 are SAT and 328 UNSAT. This dataset was chosen because the model is small enough so that the properties can be formally verified. These formal verification methods do not calculate the value of I, but rather confirm the existence of a counterexample for which s(x) > 0.We ran our approach on all 500 properties, setting ρ = 0.1, N = 10 4, M = 1000 (the choice of these hyperparameters will be justified in the next subsection), and using a uniform distribution over the input constraints as the perturbation model, along with a uniform random walk proposal. We compared our metric estimation approach against the naive Monte Carlo estimate using 10 10 samples. The generated estimates of I for all SAT properties are shown in FIG0.Both our approach and the naive MC baseline correctly identified all of the UNSAT properties by estimating I as exactly zero. However, despite using substantially more samples, naive MC failed to find a counterexample for 8 of the rarest SAT properties, thereby identifying them as UNSAT, whereas our approach found a counterexample for all the SAT properties. As shown in FIG0, the variances in the estimates for I of our approach were also very low and matched the unbiased MC baseline estimates for the more commonly violated properties, for which the latter approach still gives reliable, albeit less efficient, estimates. Along with the improved ability to predict rare events, our approach was also significantly faster than naive MC throughout, with a speed up of several orders of magnitude for properties where the event is not rare-a single run with naive MC took about 3 minutes, whereas a typical run of ours took around 3 seconds. As demonstrated by BID0, AMLS is unbiased under the assumption that perfect sampling from the targets, {π k} K−1 k=1, is possible, and that the cumulative distribution function of s(X) is continuous. In practice, finite mixing rates of the Markov chains and the dependence between the Error bars indicating ± three standard errors from 30 runs are included here and throughout, but the variance of the estimates was so small that these are barely visible. We can further conclude low bias of our method for the properties where naive MC estimation was feasible, due to the fact that naive MC produces unbiased (but potentially high variance) estimates. (b) Mean AMLS estimate relative to naive MC estimate for different ρ holding M = 1000 fixed, for those properties with log 10 I > −6.5 such that they could be estimated accurately. The bias decreases both as ρ and the rareness of the event decrease. (c) As per (b) but with varying M and holding ρ = 0.1 fixed.initialization points for each target means that sampling is less than perfect, but improves with larger values of M and N. The variance, on the other hand, theoretically strictly decreases with larger values of N and ρ BID0. In practice, we found that while larger values of M and N were always beneficial, setting ρ too high introduced biases into the estimate, with ρ = 0.1 empirically providing a good trade-off between bias and variance. Furthermore, this provides faster run times than large values of ρ, noting that the smaller values of ρ lead to larger gaps in the levels. To investigate the effect of the parameters more formally, we further ran AMLS on the SAT properties of COLLISIONDETECTION, varying ρ ∈ {0.1, 0.25, 0.5}, N ∈ {10 3, 10 4, 10 5} and M ∈ {100, 250, 1000}, again comparing to the naive MC estimate for 10 10 samples. We found that the value of N did not make a discernible difference in this range regardless of the values for ρ and M, and thus all presented correspond to setting N = 10 4. As shown in FIG0, we found that the setting of ρ made a noticeable difference to the estimates for the relatively rarer events. All the same, these differences were small relative to the differences between properties. As shown in FIG0, the value of M made little difference when ρ = 0.1,. Interesting though, we found that the value of M was important for different values of ρ, as shown in Appendix C.1, with larger values of M giving better as expected. To validate the algorithm on a higher-dimensional problem, we first tested adversarial properties on the MNIST and CIFAR-10 datasets using a dense ReLU network with two hidden-layer of size 256. An l ∞ -norm ball perturbation around the data point with width was used as the uniform input model, with = 1 representing an l ∞ -ball filling the entire space (the pixels are scaled to), together with a uniform random walk MH proposal. After training the classifiers, multilevel splitting was run on ten samples from the test set at multiple values of, with N = 10000 and ρ = 0.1, and M ∈ {100, 250, 1000} for MNIST and M ∈ {100, 250, 500, 1000, 2000} for CIFAR-10. The for naive MC were also evaluated using 5 × 10 9 samples-less than the previous experiment as the larger network made estimation more expensive-in the cases where the event was not too rare. This took around twenty minutes per naive MC estimate, versus a few minutes for each AMLS estimate. As the were similar across datapoints, we present the for a single example in the top two rows of FIG1. As desired, a smooth curve is traced out as decreases, for which the event E becomes rarer. For MNIST, acceptable accuracy is obtained for M = 250 and high accuracy for M = 1000. For CIFAR-10, which has about four times the input dimension of MNIST, larger values of M were required to achieve comparable accuracy. The magnitude of required to give a certain value of log(I) is smaller for CIFAR-10 than MNIST, reflecting that adversarial examples for the former are typically more perceptually similar to the datapoint. Estimates for I on adversarial properties of a single datapoint with ρ = 0.1, and N ∈ {10000, 10000, 300} for MNIST/CIFAR-10/CIFAR-100 respectively. As in FIG0, the error bars from 30 runs are barely visible, highlighting a very low variance in the estimates, while the close matching to the naive MC estimates when is large enough to make the latter viable, indicate a very low bias. For CIFAR-100 the error bars are shown for the naive estimates, as well, from 10 runs.[Right] The difference in the estimate for the other values of M from M ∈ {1000, 2000, 2000} for MNIST/CIFAR-10/CIFAR-100, respectively. The estimate steadily converges as M increases, with larger M more important for rarer events. To demonstrate that our approach can be employed on large networks, we tested adversarial properties on the CIFAR-100 dataset and a much larger DenseNet architecture BID10, with depth and growth-rate 40 (approximately 2 × 10 6 parameters). Due to the larger model size, we set N = 300, the largest minibatch that could be held in memory (a larger N could be used by looping over minibatches). The naive Monte Carlo estimates used 5 × 10 6 samples for about an hour of computation time per estimate, compared to between five to fifteen minutes for each AMLS estimate. The are presented in the bottom row of FIG1, showing that our algorithm agrees with the naive Monte Carlo estimate. We now examine how our robustness metric varies for a ReLU network as that network is trained to be more robust against norm bounded perturbations to the inputs using the method of BID27. Roughly speaking, their method works by approximating the set of outputs ing from perturbations to an input with a convex outer bound, and minimizing the worst case loss over this FORMULA0 produces a certificate-of-robustness for = 0.1 ("W&K"), versus the fraction of those samples for which I = P min for ∈ {0.1, 0.2, 0.3} ("AMLS"). Due to very heavy memory requirements, it was computationally infeasible to calculate certificates-of-robustness for = {0.2, 0.3}, and = 0.1 before epoch 32 with the method of BID27. Our metric, however, suffers no such memory issues.set. The motivation for this experiment is twofold. Firstly, this training provides a series of networks with ostensibly increasing robustness, allowing us to check if our approach produces robustness estimates consistent with this improvement. Secondly, it allows us to investigate whether the training to improve robustness for one type of adversarial attack helps to protect against others. Specifically, whether training for small perturbation sizes improves robustness to larger perturbations. We train a CNN model on MNIST for 100 epochs with the standard cross-entropy loss, then train the network for a further 100 epochs using the robust loss of BID27, saving a snapshot of the model at each epoch. The architecture is the same as in BID27, containing two strided convolutional layers with 16 and 32 channels, followed by two fully connected layers with 100 and 10 hidden units, and ReLU activations throughout. The robustification phase trains the classifier to be robust in an l ∞ -ball around the inputs, where is annealed from 0.01 to 0.1 over the first 50 epochs. At a number of epochs during the robust training, we calculate our robustness metric with ∈ {0.1, 0.2, 0.3} on 50 samples from the test set. The are summarized in FIG2 with additional per-sample in Appendix C.2. We see that our approach is able to capture variations in the robustnesses of the network. As the method of BID27 returns the maximum value of the property for each sample over a convex outer bound on the perturbations, it is able to produce certificates-of-robustness for some datapoints. If the returned is less than 0 then no adversarial examples exist in an l ∞ ball of radius around that datapoint. If the returned is greater than 0, then the datapoint may or may not be robust in that l ∞ ball, due to fact that it optimizes over an outer bound. Though we emphasize that the core aim of our approach is in providing richer information for SAT properties, this provides an opportunity to see how well it performs at establishing UNSAT properties relative to a more classical approach. To this end, we compared the fraction of the 50 samples from the test set that are verified by the method of BID27, to the fraction that have a negligible volume of adversarial examples, I = P min, in their l ∞ -ball neighbourhood. The are presented in FIG2.Our method forms an upper bound on the fraction of robust samples, which can be made arbitrarily tighter by taking P min → 0. BID27, on the other hand, forms a lower bound on the fraction of robust samples, where the tightness of the bound depends on the tightness of the convex outer bound, which is unknown and cannot be controlled. Though the true value must lie somewhere between the two bounds, our bound still holds physical meaning it its own right in a way that BID27 does not: it is the proportion of samples for which the prevalence of violations is less than an a given acceptable threshold P min.This experiment also highlights an important shortcoming of BID27. The memory usage of their procedure depends on how many ReLU activations cross their threshold over perturbations. This is high during initial training for = 0.1 and indeed the reason why the training procedure starts from = 0.01 and gradually anneals to = 0.1. The is that it is infeasible (the GPU memory is exhausted)-even for this relatively small model-to calculate the maximum value of the property on the convex outer bound for ∈ {0.2, 0.3} at all epochs, and = 0.1 for epochs before 32. Even in this restricted setting where our metric has been reduced to a binary one, it appears to be more informative than that of BID27 for this reason. We have introduced a new measure for the intrinsic robustness of a neural network, and have validated its utility on several datasets from the formal verification and deep learning literatures. Our approach was able to exactly emulate formal verification approaches for satisfiable properties and provide high confidence, accurate predictions for properties which were not. The two key advantages it provides over previous approaches are: a) providing an explicit and intuitive measure for how robust networks are to satisfiable properties; and b) providing improved scaling over classical approaches for identifying unsatisfiable properties. Despite providing a more informative measure of how robust a neural network is, our approach may not be appropriate in all circumstances. In situations where there is an explicit and effective adversary, instead of inputs being generated by chance, we may care more about how far away the single closest counterexample is to the input, rather than the general prevalence of counterexamples. Here our method may fail to find counterexamples because they reside on a subset with probability less than P min; the counterexamples may even reside on a subset of the input space with measure zero with respect to the input distribution. On the other hand, there are many practical scenarios, such as those discussed in the introduction, where either it is unrealistic for there to be no counterexamples close to the input, the network (or input space) is too large to realistically permit formal verification, or where potential counterexamples are generated by chance rather than by an adversary. We believe that for these scenarios our approach offers significant advantages to formal verification approaches. Going forward, one way the efficiency of our approach could be improved further is by using a more efficient base MCMC kernel in our AMLS estimator, that is, replace line 12 in Algorithm 1 with a more efficient base inference scheme. The current MH scheme was chosen on the basis of simplicity and the fact it already gave effective empirical performance. However, using more advanced inference approaches, such as gradient-based approaches like Langevin Monte Carlo (LMC) BID21 Hamiltonian Monte Carlo , could provide significant speedups by improving the mixing of the Markov chains, thereby reducing the number of required MCMC transitions. We gratefully acknowledge Sebastian Nowozin for suggesting to us to apply multilevel splitting to the problem of estimating neural network robustness. We also thank Rudy Bunel for his help with the COLLISIONDETECTION dataset, and Leonard Berrada for supplying a pretrained DenseNet model. Metropolis-Hastings (MH) is an MCMC method that allows for sampling when one only has access to an unnormalized version of the target distribution BID6. At a high-level, one attempts iteratively proposes local moves from the current location of a sampler and then accepts or rejects this move based on the unnormalized density. Each iteration of this process is known as a MH transition. The unnormalized targets distributions of interest for our problem are γ k (x) where DISPLAYFORM0 A MH transition now consists of proposing a new sample using a proposal x ∼ g(x | x), where x indicates the current state of the sampler and x the proposed state, calculating an acceptance probability, DISPLAYFORM1 and accepting the new sample with probability A k (x | x), returning the old sample if the new one is rejected. The proposal, g(x | x), is a conditional distribution, such as a normal distribution centred at x with fixed covariance matrix. Successive applications of this transition process generates samples which converge in distribution to the target π k (x) and whose correlation with the starting sample diminishes to zero. In our approach, these MH steps are applied independently to each sample in the set, while the only samples used for the AMLS algorithm are the final samples produced from the ing Markov chains. Algorithm 1 has computational cost O(N M K), where the number of levels K will depend on the rareness of the event, with more computation required for rarer ones. Parallelization over N is possible provided that the batches fit into memory, whereas the loops over M and K must be performed sequentially. One additional change we make from the approach outlined by BID9 is that we perform MH updates on all chains in Lines 12, rather than only those that were previously killed off. This helps reduce the build up of correlations over multiple levels, improving performance. Another is that we used an adaptive scheme for g(x |x) to aid efficiency. Specifically, our proposal takes the form of a random walk, the radius of which,, is adapted to keep the acceptance ratio roughly around 0.234 (see BID20). Each chain has a separate acceptance ratio that is average across MH steps, and after M MH steps, for those chains whose acceptance ratio is below 0.234 it is halved, and conversely for those above 0.234, multiplied by 1.02. DISPLAYFORM0 Whereas the exact value of M within the range considered proved to not be especially important when ρ = 0.1, it transpires to have a large impact in the quality of the for larger values of ρ as shown in Figure 4.C.2 PER-SAMPLE ROBUSTNESS MEASURE DURING ROBUST TRAINING FIG5 illustrates the diverse forms that the per-sample robustness measure can take on the 40 datapoints averaged over in Experiment §5.3. We see that different datapoints have quite varying initial levels of robustness, and that the training helps with some points more than others. In one case, the datapoint was still not robust add the end of training for the target perturbation size = 0.1. Figure 4: Mean AMLS estimate relative to naive (unbiased) MC estimate for different M = holding ρ fixed to 0.25 (left) and 0.5 (right), for those properties whose naive MC estimate was greater than log 10 I = −6.5 such that they could be estimated accurately.

Combining auxiliary and adversarial training to interrogate and help physical understanding.

Represent the TLDR for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Flow based models such as Real NVP are an extremely powerful approach to density estimation. However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables. This makes them poorly suited for modeling and representing discrete structures in data distributions, for example class membership or discrete symmetries. To address this difficulty, we present a normalizing flow architecture which relies on domain partitioning using locally invertible functions, and possesses both real and discrete valued latent variables. This Real and Discrete (RAD) approach retains the desirable normalizing flow properties of exact sampling, exact inference, and analytically computable probabilities, while at the same time allowing simultaneous modeling of both continuous and discrete structure in a data distribution. Latent generative models are one of the prevailing approaches for building expressive and tractable generative models. The generative process for a sample x can be expressed as DISPLAYFORM0 where z is a noise vector, and g a parametric generator network (typically a deep neural network). This paradigm has several incarnations, including variational autoencoders , generative adversarial networks , and flow based models BID0 BID9 BID5; BID3 ).The training process and model architecture for many existing latent generative models, and for all published flow based models, assumes a unimodal smooth distribution over latent variables z. Given the parametrization of g as a neural network, the mapping to x is a continuous function. This imposed structure makes it challenging to model data distributions with discrete structure -for instance, multi-modal distributions, distributions with holes, distributions with discrete symmetries, or distributions that lie on a union of manifolds (as may approximately be true for natural images, see BID11 . Indeed, such cases require the model to learn a generator whose input Jacobian has highly varying or infinite magnitude to separate the initial noise source into different clusters. Such variations imply a challenging optimization problem due to large changes in curvature. This shortcoming can be critical as several problems of interest are hypothesized to follow a clustering structure, i.e. the distributions is concentrated along several disjoint connected sets .A standard way to address this issue has been to use mixture models BID16; ) or structured priors . In order to efficiently parametrize the model, mixture models are often formulated as a discrete latent variable models (; BID4 ; van den Oord model (1c, 1d). Note the dependency of K on Z in 1d. While this is not necessary, we will exploit this structure as highlighted later in the main text and in Figure 4. et al., 2017), some of which can be expressed as a deep mixture model BID10 BID14 BID13. Although the ing exponential number of mixture components with depth in deep mixture models is an advantage in terms of expressivity, it is an impediment to inference, evaluation, and training of such models, often requiring as a the use of approximate methods like hard-EM or variational inference .In this paper we combine piecewise invertible functions with discrete auxiliary variables, selecting which invertible function applies, to describe a deep mixture model. This framework enables a probabilistic model's latent space to have both real and discrete valued units, and to capture both continuous and discrete structure in the data distribution. It achieves this added capability while preserving the exact inference, exact sampling, exact evaluation of log-likelihood, and efficient training that make standard flow based models desirable. We aim to learn a parametrized distribution p X (x) on the continuous input domain R d by maximizing log-likelihood. The major obstacle to training an expressive probabilistic model is typically efficiently evaluating log-likelihood. If we consider a mixture model with a large number |K| of components, the likelihood takes the form DISPLAYFORM0 In general, evaluating the likelihood requires computing probabilities for all |K| components. However, following a strategy similar to , if we partition the domain DISPLAYFORM1, we can write the likelihood as DISPLAYFORM2 This transforms the problem of summation to a search problem x → f K (x). This can be seen as the inferential converse of a stratified sampling strategy BID8. The proposed approach will be a direct extension of flow based models BID6 BID5 ). Flow based models enable log-likelihood (a) An example of a trimodal distribution pX, sinusoidal distribution. The different modes are colored in red, green, and blue.(b) The ing unimodal distribution pZ, corresponding to the distribution of any of the initial modes in pX.(c) An example fZ (x) of a piecewise invertible function aiming at transforming pZ into a unimodal distribution. The red, green, and blue zones corresponds to the different modes in input space. Figure 2: Example of a trimodal distribution (2a) turned into a unimodal distribution (2b) using a piecewise invertible function (2c). Note that the initial distribution p X correspond to an unfolding of DISPLAYFORM0 evaluation by relying on the change of variable formula DISPLAYFORM1 with f Z a parametrized bijective function from R d onto R d and ∂f Z ∂x T the absolute value of the determinant of its Jacobian. As also proposed in , we relax the constraint that f Z be bijective, and instead have it be surjective onto R d and piecewise invertible. That is, we require f Z|A k (x) be an invertible function, where DISPLAYFORM2 we can define the following generative process: DISPLAYFORM3 If we use the set identification function f K associated with A k, the distribution corresponding to this stochastic inversion can be defined by a change of variable formula DISPLAYFORM4 Because of the use of both Real and Discrete stochastic variables, we call this class of model RAD.The particular parametrization we use on is depicted in Figure 2. We rely on piecewise invertible functions that allow us to define a mixture model of repeated symmetrical patterns, following a Figure 4: Illustration of the expressive power the gating distribution p K|Z provides. By capturing the structure in a sine wave in p K|Z, the function z, k → x can take on an extremely simple form, corresponding only to a linear function with respect to z. DISPLAYFORM5 method of folding the input space. Note that in this instance the function f K is implicitly defined by f Z, as the discrete latent corresponds to which invertible component of the piecewise function x falls on. So far, we have defined a mixture of |K| components with disjoint support. However, if we factorize p Z,K as p Z · p K|Z, we can apply another piecewise invertible map to Z to define p Z as another mixture model. Recursively applying this method in a deep mixture model (see FIG2).Another advantage of such factorization is in the gating network p K|Z, as also designated in (van den BID13 . It provides a more constrainted but less sample wasteful approach than rejection sampling BID1 by taking into account the untransformed sample z before selecting the mixture component k. This allows the model to exploit the distribution p Z in different regions A k in more complex ways than repeating it as a patternm as illustrated in Figure 4 .The function from the input to the discrete variables, f K (x), contains discontinuities. This presents the danger of introducing discontinuities into log p X (x), making optimization more difficult. However, by carefully imposing boundary conditions on the gating network, we are able to exactly counteract the effect of discontinuities in f K, and cause log p X (x) to remain continuous with respect to the parameters. This is discussed in detail in Appendix A. We conduct a brief comparison on six two-dimensional toy problems with REAL NVP to demonstrate the potential gain in expressivity RAD models can enable. Synthetic datasets of 10, 000 points each are constructed following the manifold hypothesis and/or the clustering hypothesis. We designate these problems as: grid Gaussian mixture, ring Gaussian mixture, two moons, two circles, spiral, and many moons (see FIG3). For the RAD model implementation, we use the piecewise linear activations defined in Appendix A in a coupling layer architecture BID5 for f Z where, instead of a conditional linear transformation, the conditioning variable x 1 determines the parameters of the piecewise linear activation on x 2 to obtain z 2 and k 2, with z 1 = x 1 (see FIG4). For the gating network p K|Z, the gating logit neural network s (z) take as input z = (z 1, z 2). We compare with a REAL NVP model using only affine coupling layers. p Z is a standard Gaussian distribution. As both these models can easily approximately solve these generative modeling tasks provided enough capacity, we study these model in a relatively low capacity regime, where we can showcase the potential expressivity RAD may provide. Each of these models uses six coupling layers, and each coupling layer uses a one-hidden-layer rectified network with a tanh output activation scaled by a scalar parameter as described in. For RAD, the logit network s (·) also uses a one-hidden-layer rectified neural network, but with linear output. In order to fairly compare with respect to number of parameters, we provide REAL NVP seven times more hidden units per (e) REAL NVP on spiral.(f) REAL NVP on many moons.(g) RAD on grid Gaussian mixture.(h) RAD on ring Gaussian mixture.(i) RAD on two moons.(j) RAD on two circles.(k) RAD on spiral.(l) RAD on many moons. Figure 7: Comparison of samples from trained REAL NVP (top row) (a-f) and RAD (bottow row) (g-l) models. REAL NVP fails in a low capacity setting by attributing probability mass over spaces where the data distribution has low density. Here, these spaces often connect data clusters, illustrating the challenges that come with modeling multimodal data as one continuous manifold.hidden layer than RAD, which uses 8 hidden units per hidden layer. For each level, p K|Z and f Z are trained using stochastic gradient ascent with ADAM on the log-likelihood with a batch size of 500 for 50, 000 steps. In each of these problems, RAD is consistently able to obtain higher log-likelihood than REAL NVP. We plot the samples (Figure 7) of the described RAD and REAL NVP models trained on these problems. In the described low capacity regime, REAL NVP fails by attributing probability mass over spaces where the data distribution has low density. This is consistent with the mode covering behavior of maximum likelihood. However, the particular inductive bias of REAL NVP is to prefer modeling the data as one connected manifold. This in the unwanted probability mass being distributed along the space between clusters. Flow-based models often follow the principle of Gaussianization , i.e. transforming the data distribution into a Gaussian. The inversion of that process on a Gaussian distribution would therefore approximate the data distribution. We plot in FIG7 the inferred Gaussianized variables z for both models trained on the ring Gaussian mixture problem. The Gaussianization from REAL NVP leaves some area of the standard Gaussian distribution unpopulated. These unattended areas correspond to unwanted regions of probability mass in the input space. RAD suffers significantly less from this problem. An interesting feature is that RAD seems also to outperform REAL NVP on the spiral dataset. One hypothesis is that the model successfully exploits some non-linear symmetries in this problem. In RAD several points which were far apart in the input space become neighbors in z. This is not the case for REAL NVP. We take a deeper look at the Gaussianization process involved in both models. In FIG8 we plot the inference process of z from x for both models trained on the two moons problem. As a of a folding process similar to that in , several points which were far apart in the input space become neighbors in z in the case of RAD.We further explore this folding process using the visualization described in FIG1. We verify that the non-linear folding process induced by RAD plays at least two roles: bridging gaps in the distribution of probability mass, and exploiting symmetries in the data. We observe that in the case of the ring Gaussian mixture FIG1 ), RAD effectively uses foldings in order to bridge the different modes of the distribution into a single mode, primarily in the last layers of the transformation. We contrast this with REAL NVP FIG1 ) which struggles to combine these modes under the standard Gaussian distribution using bijections. In the spiral problem FIG1 ), RAD decomposes the spiral into three different lines to bridge FIG1 ) instead of unrolling the manifold fully, which REAL NVP struggles to do FIG1 ).In both cases, the points remain generally well separated by labels, even after being pushed through a RAD layer FIG1. This enables the model to maximize the conditional log-probability log(p K|Z). We introduced an approach to tractably evaluate and train deep mixture models using piecewise invertible maps as a folding mechanism. This allows exact inference, exact generation, and exact evaluation of log-likelihood, avoiding many issues in previous discrete variables models. This method can easily be combined with other flow based architectural components, allowing flow based models to better model datasets with discrete as well as continuous structure. Figure 11: RAD and REAL NVP inference processes on the ring Gaussian mixture problem. Each column correspond to a RAD or affine coupling layer. RAD effectively uses foldings in order to bridge the multiple modes of the distribution into a single mode, primarily in the last layers of the transformation, whereas REAL NVP struggles to bring together these modes under the standard Gaussian distribution using continuous bijections. A CONTINUITYThe standard approach in learning a deep probabilistic model has been stochastic gradient descent on the negative log-likelihood. Although the model enables the computation of a gradient almost everywhere, the log-likelihood is unfortunately discontinuous. Let's decompose the log-likelihood DISPLAYFORM0 There are two sources of discontinuity in this expression: f K is a function with discrete values (therefore discontinuous) and ∂f Z ∂x T is discontinuous because of the transition between the subsets A k, leading to the expression of interest DISPLAYFORM1 which takes a role similar to the log-Jacobian determinant, a pseudo log-Jacobian determinant. Let's build from now on the simple scalar case and a piecewise linear function DISPLAYFORM2 In this case, s(z) = log p K|Z k | z k≤N + C1 |K| can be seen as a vector valued function. We can attempt at parametrizing the model such that the pseudo log-Jacobian determinant becomes continuous with respect to β by expressing the boundary condition at x = β DISPLAYFORM3 ⇒s(−α 2 β) 2 + log(α 2) = s(−α 2 β) 3 + log(α 3). DISPLAYFORM4 − log(α 1), log(α 2), log(α 3) + β 2 1 + cos (zα DISPLAYFORM5 Another type of boundary condition can be found at between the non-invertible area and the invertible area z = α 2 β, as ∀z > α 2 β, p 3|Z (3 | z) = 1, therefore DISPLAYFORM6 Since the condition ∀k < 3, p K|Z k | z) → 0 when z → (α 2 β) − will lead to an infinite loss barrier at x = −β, another way to enforce this boundary condition is by adding linear pieces FIG1 ): DISPLAYFORM7 The inverse is defined as DISPLAYFORM8 In order to know the values of s at the boundaries ±α 2 β, we can use the logit function DISPLAYFORM9 Given those constraints, the model can then be reliably learned through gradient descent methods. Note that the ing tractability of the model from the fact that the discrete variables k is only interfaced during inference with the distribution p K|Z, unlike discrete variational autoencoders approaches (; BID15 where it is fed to a deep neural network. Similar to BID7, the learning of discrete variables is achieved by relying on the the continuous component of the model, and, as opposed as other approaches (; BID12 ; BID12, this gradient signal extracted is exact and closed form. We plot the remaining inference processes of RAD and REAL NVP on the remaining problems not plotted previously: grid Gaussian mixture FIG1, two circles FIG1), two moons FIG1, and many moons FIG1 ). We also compare the final of the Gaussianization processes on both models on the different toy problems in FIG1. (e) REAL NVP on spiral.(f) REAL NVP on many moons.(g) RAD on grid Gaussian mixture.(h) RAD on ring Gaussian mixture.(i) RAD on two moons.(j) RAD on two circles.(k) RAD on spiral.(l) RAD on many moons. FIG1: Comparison of the Gaussianization from the trained REAL NVP (top row) (a-f) and RAD (bottow row) (g-l). REAL NVP fails in a low capacity setting by leaving unpopulated areas where the standard Gaussian attributes probability mass. Here, these spaces as often ones separating clusters, showing the failure in modeling the data as one manifold.

Flow based models, but non-invertible, to also learn discrete variables

Represent the short summary for finding the corresponding scientific article (long)

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.