Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance. Deep neural networks have revolutionized many areas of machine intelligence and are now used for many vision tasks that even few years ago were considered nearly impenetrable BID15 BID26. Advances in neural networks and hardware is ing in much of the computation being shifted to consumer devices, delivering faster response, and better security and privacy guarantees BID11 BID8.As the space of deep learning applications expands and starts to personalize, there is a growing need for the ability to quickly build and customize models. While model sizes have dropped dramatically from >50M parameters of the pioneering work of AlexNet BID15 and VGG BID26 to <5M of the recent Mobilenet BID25 BID8 and ShuffleNet BID30 BID19, the accuracy of models has been improving. However, delivering, maintaining and updating hundreds of models on the embedded device is still a significant expense in terms of bandwidth, energy and storage costs. While there still might be space for improvement in designing smaller models, in this paper we explore a different angle: we would like to be able to build models that require only a few parameters to be trained in order to be re-purposed to a different task, with minimal loss in accuracy compared to a model trained from scratch. While there is ample existing work on compressing models and learning as few weights as possible BID24 BID25 BID8 to solve a single task, to the best of our awareness, there is no prior work that tries to minimize the number of model parameters when solving many tasks together. Our contribution is a novel learning paradigm in which each task carries its own model patcha small set of parameters -that, along with a shared set of parameters constitutes the model for that task (for a visual description of the idea, see FIG0, left side). We put this idea to use in two scenarios: a) in transfer learning, by fine-tuning only the model patch for new tasks, and b) in multi-task learning, where each task performs gradient updates to both its own model patch, and the shared parameters. In our experiments (Section 5), the largest patch that we used is smaller than 10% of the size of the entire model. We now describe our contribution in detail. Transfer learning We demonstrate that, by fine-tuning less than 35K parameters in MobilenetV2 BID25 and InceptionV3, our method leads to significant accuracy improvements over fine-tuning only the last layer (102K-1.2M parameters, depending on the number of classes) on multiple transfer learning tasks. When combined with fine-tuning the last layer, we train less than 10% of the model's parameters in total. We also show the effectiveness of our method over last-layer-based fine-tuning on transfer learning between completely different problems, namely COCO-trained SSD model to classification over ImageNet BID4.Multi-task learning We explore a multi-task learning paradigm wherein multiple models that share most of the parameters are trained simultaneously (see FIG0, right side). Each model has a task-specific model patch. Training is done in a distributed manner; each task is assigned a subset of available workers that send independent gradient updates to both shared and task-specific parameters using standard optimization algorithms. Our show that simultaneously training two such MobilenetV2 BID25 ) models on ImageNet BID4 ) and Places-365 reach accuracies comparable to, and sometimes higher than individually trained models. We apply our multi-task learning paradigm to domain adaptation. For ImageNet BID4, we show that we can simultaneously train MobilenetV2 BID25 ) models operating at 5 different resolution scales, 224, 192, 160, 128 and 96, while sharing more than 98% of the parameters and ing in the same or higher accuracy as individually trained models. This has direct practical benefit in power-constrained operation, where an application can switch to a lower resolution to save on latency/power, without needing to ship separate models and having to make that trade-off decision at the application design time. The cascade algorithm from BID27 can further be used to reduce the average running time by about 15% without loss in accuracy. The rest of the paper is organized as follows: we describe our method in Section 2 and discuss related work in Section 3. In Section 4, we present simple mathematical intuition that contrasts the expressiveness of logit-only fine-tuning and that of our method. Finally, in Section 5, we present detailed experimental . The central concept in our method is that of a model patch. It is essentially a small set of perchannel transformations that are dispersed throughout the network ing in only a tiny increase in the number of model parameters. Suppose a deep network M is a sequence of layers represented by their parameters (weights, biases), W 1,..., W n. We ignore non-trainable layers (e.g., some kinds of activations) in this formulation. A model patch P is a set of parameters W i1,..., W i k, 1 ≤ i 1,..., i k ≤ n that, when applied to M, adds layers at positions i 1,..., i n. Thus, a patched model In this paper, we introduce two kinds of patches. We will see below that they can be folded with the other layers in the network, eliminating the need to perform any explicit addition of layers. In Section 5, we shed some light on why the particular choice of these patches is important. Scale-and-bias patch This patch applies per-channel scale and bias to every layer in the network. In practice this transformations can often be absorbed into normalization layer such as Batch Normalization BID9. Let X be an activation tensor. Then, the batch-normalized version of X DISPLAYFORM0 where µ(X), σ(X) are mean and standard deviation computed per minibatch, and γ, β are learned via backpropagation. These statistics are computed as mini-batch average, while during inference they are computed using global averages. The scale-and-bias patch corresponds to all the γ, β, µ, σ in the network. Using BN as the model patch also satisfies the criterion that the patch size should be small. For instance, the BN parameters in both MobilenetV2 BID25 and InceptionV3 network performing classification on ImageNet amounts to less than 40K parameters, of about 1% for MobilenetV2 that has 3.5 million Parameters, and less than 0.2% for Inception V3 that has 25 million parameters. While we utilize batch normalization in this paper, we note that this is merely an implementation detail and we can use explicit biases and scales with similar . Depthwise-convolution patch The purpose of this patch is to re-learn spatial convolution filters in a network. Depth-wise separable convolutions were introduced in deep neural networks as way to reduce number of parameters without losing much accuracy BID8 BID2. They were further developed in BID25 by adding linear bottlenecks and expansions. In depthwise separable convolutions, a standard convolution is decomposed into two layers: a depthwise convolution layer, that applies one convolutional filter per input channel, and a pointwise layer that computes the final convolutional features by linearly combining the depthwise convolutional layers' output across channels. We find that the set of depthwise convolution layers can be repurposed as a model patch. They are also lightweight -for instance, they account for less than 3% of MobilenetV2's parameters when training on ImageNet. Next, we describe how model patches can be used in transfer and multi-task learning. Transfer learning In transfer learning, the task is to adapt a pretrained model to a new task. Since the output space of the new task is different, it necessitates re-learning the last layer. Following our approach, we apply a model patch and train the patched parameters, optionally also the last layer. The rest of the parameters are left unchanged. In Section 5, we discuss the inclusion/exclusion of the last layer. When the last layer is not trained, it is fixed to its random initial value. Multitask learning We aim to simultaneously, but independently, train multiple neural networks that share most weights. Unlike in transfer learning, where a large fraction of the weights are kept frozen, here we learn all the weights. However, each task carries its own model patch, and trains a patched model. By training all the parameters, this setting offers more adaptability to tasks while not compromising on the total number of parameters. To implement multi-task learning, we use the distributed TensorFlow paradigm 1: a central parameter server receives gradient updates from each of the workers and updates the weights. Each worker reads the input, computes the loss and sends gradients to the parameter server. We allow subsets of workers to train different tasks; workers thus may have different computational graphs, and taskspecific input pipelines and loss functions. A visual depiction of this setting is shown in FIG0. One family of approaches BID29 BID5 widely used by practitioners for domain adaptation and transfer learning is based on fine-tuning only the last layer (or sometimes several last layers) of a neural network to solve a new task. Fine-tuning the last layer is equivalent to training a linear classifier on top of existing features. This is typically done by running SGD while keeping the rest of the network fixed, however other methods such as SVM has been explored as well BID10. It has been repeatedly shown that this approach often works best for similar tasks (for example, see BID5).Another frequently used approach is to use full fine-tuning BID3 ) where a pretrained model is simply used as a warm start for the training process. While this often leads to significantly improved accuracy over last-layer fine-tuning, downsides are that 1) it requires one to create and store a full model for each new task, and 2) it may lead to overfitting when there is limited data. In this work, we are primarily interested in approaches that allow one to produce highly accurate models while reusing a large fraction of the weights of the original model, which also addresses the overfitting issue. While the core idea of our method is based on learning small model patches, we see significant boost in performance when we fine-tune the patch along with last layer (Section 5). This is somewhat in contrast with BID7, where the authors show that the linear classifier (last layer) does not matter when training full networks. Mapping out the conditions of when a linear classifier can be replaced with a random embedding is an important open question. BID16 show that re-computing batch normalization statistics for different domains helps to improve accuracy. In BID24 it was suggested that learning batch normalization layers in an otherwise randomly initialized network is sufficient to build non-trivial models. Recomputing batch normalization statistics is also frequently used for model quantization where it prevents the model activation space from drifting BID13. In the present work, we significantly broaden and unify the scope of the idea and scale up the approach by performing transfer and multi-task learning across completely different tasks, providing a powerful tool for many practical applications. Our work has interesting connections to meta-learning BID22 BID6. For instance, when training data is not small, one can allow each task to carry a small model patch in the Reptile algorithm of BID22 in order to increase expressivity at low cost. Experiments (Section 5) show that model-patch based fine-tuning, especially with the scale-andbias patch, is comparable and sometimes better than last-layer-based fine-tuning, despite utilizing a significantly smaller set of parameters. At a high level, our intuition is based on the observation that individual channels of hidden layers of neural network form an embedding space, rather than correspond to high-level features. Therefore, even simple transformations to the space could in significant changes in the target classification of the network. In this section (and in Appendix A), we attempt to gain some insight into this phenomenon by taking a closer look at the properties of the last layer and studying low-dimensional models. A deep neural network performing classification can be understood as two parts:1. a network base corresponding to a function F: R d → R n mapping d-dimensional input space X into an n-dimensional embedding space G, and 2. a linear transformation s: R n → R k mapping embeddings to logits with each output component corresponding to an individual class. DISPLAYFORM0 We compare fine-tuning model patches with fine-tuning only the final layer s. Fine-tuning only the last layer has a severe limitation caused by the fact that linear transformations preserve convexity. It is easy to see that, regardless of the details of s, the mapping from embeddings to logits is such that if both ξ a, ξ b ∈ G are assigned label c, the same label is assigned to every DISPLAYFORM1 Thus, if the model assigns inputs {x i |i = 1, . . ., n c} some class c, then the same class will also be assigned to any point in the preimage of the convex hull of {F (x i)|i = 1,..., n c }.This property of the linear transformation s limits one's capability to tune the model given a new input space manifold. For instance, if the input space is "folded" by F and the neighborhoods of very different areas of the input space X are mapped to roughly the same neighborhood of the embedding space, the final layer cannot disentangle them while operating on the embedding space alone (should some new task require differentiating between such "folded" regions).We illustrate the difference in expressivity between model-patch-based fine-tuning and last-layerbased fine-tuning in the cases of 1D (below) and 2D (Appendix A) inputs and outputs. Despite the simplicity, our analysis provides useful insights into how by simply adjusting biases and scales of a neural network, one can change which regions of the input space are folded and ultimately the learned classification function. In what follows, we will work with a construct introduced by BID21 that demonstrates how neural networks can "fold" the input space X a number of times that grows exponentially with the neural network depth 2. We consider a simple neural network with one-dimensional inputs and outputs and demonstrate that a single bias can be sufficient to alter the number of "folds", the topology of the X → G mapping. More specifically, we illustrate how the number of connected components in the preimage of a one-dimensional segment [ξ a, ξ b] can vary depending on a value of a single bias variable. As in BID21, consider the following function: DISPLAYFORM2 p is an even number, and b = (b 0, . . ., b p) is a (p + 1)-dimensional vector of tunable parameters characterizing q. Function q(x; b) can be represented as a two-layer neural network with ReLU activations. Set p = 2. Then, this network has 2 hidden units and a single output value, and is capable of "folding" the input space twice. Defining F to be a composition of k such functions DISPLAYFORM3 we construct a neural network with 2k layers that can fold input domain R up to 2 k times. By 0, the number of "folds" can vary from 2 k to 0. We evaluate the performance of our method in both transfer and multi-task learning using the image recognition networks MobilenetV2 BID25 and InceptionV3 et al., 2013), Aircraft BID20, Flowers-102 BID23 and Places-365 ). An overview of these datasets can be found in TAB0. We also show preliminary on transfer learning across completely different types of tasks using MobilenetV2 and Single-Shot Multibox Detector (SSD) networks. We use both scale-and-bias (S/B) and depthwise-convolution patches (DW) in our experiments. Both MobilenetV2 and InceptionV3 have batch normalization -we use those parameters as the S/B patch. MobilenetV2 has depthwise-convolutions from which we construct the DW patch. In our experiments, we also explore the effect of fine-tuning the patches along with the last layer of the network. We compare with two scenarios: 1) only fine-tuning the last layer, and 2) fine-tuning the entire network. We use TensorFlow BID0, and NVIDIA P100 and V100 GPUs for our experiments. Following the standard setup of Mobilenet and Inception we use 224 × 224 images for MobilenetV2 and 299 × 299 for InceptionV3. As a special-case, for Places-365 dataset, we use 256 × 256 images. We use RMSProp optimizer with a learning rate of 0.045 and decay factor 0.98 per 2.5 epochs. To demonstrate the expressivity of the biases and scales, we perform an experiment on MobilenetV2, where we learn only the scale-and-bias patch while keeping the rest of the parameters frozen at their initial random state. The are shown in TAB2 (right side). It is quite striking that simply adjusting biases and scales of random embeddings provides features powerful enough that even a linear classifier can achieve a non-trivial accuracy. Furthermore, the synergy exhibited by the combination of the last layer and the scale-and-bias patch is remarkable. We take MobileNetV2 and InceptionV3 models pretrained on ImageNet (Top1 accuracies 71.8% and 76.6% respective), and fine-tune various model patches for other datasets. Results on InceptionV3 are shown in TAB1. We see that fine-tuning only the scale-and-bias patch (using a fixed, random last layer) in comparable accuracies as fine-tuning only the last layer while using fewer parameters. Compared to full fine-tuning BID3, we use orders of magnitude fewer parameters while achieving nontrivial performance. Our using MobilenetV2 are similar (more on this later).In the next experiment, we do transfer learning between completely different tasks. We take an 18-category object detection (SSD) model ) pretrained on COCO images (Lin et al., TAB2 . Again, we see the effectiveness of training the model patch along with the last layer -a 2% increase in the parameters translates to 19.4% increase in accuracy. Next, we discuss the effect of learning rate. It is common practice to use a small learning rate when fine-tuning the entire network. The intuition is that, when all parameters are trained, a large learning rate in network essentially forgetting its initial starting point. Therefore, the choice of learning rate is a crucial factor in the performance of transfer learning. In our experiments (Appendix B.2, FIG11) we observed the opposite behavior when fine-tuning only small model patches: the accuracy grows as learning rate increases. In practice, fine-tuning a patch that includes the last layer is more stable w.r.t. the learning rate than full fine-tuning or fine-tuning only the scale-and-bias patch. Finally, an overview of on MobilenetV2 with different learning rates and model patches is shown in FIG4. The effectiveness of small model patches over fine-tuning only the last layer is again clear. Combining model patches and fine-tuning in a synergistic effect. In Appendix B, we show additional experiments comparing the importance of learning custom bias/scale with simply updating batch-norm statistics (as suggested by BID16). In this section we show that, when using model-specific patches during multi-task training, it leads to performance comparable to that of independently trained models, while essentially using a single model. We simultaneously train MobilenetV2 BID25 on two large datasets: ImageNet and Places365. Although the network architecture is the same for both datasets, each model has its own private patch that, along with the rest of the model weights constitutes the model for that dataset. We choose a combination of the scale-and-bias patch, and the last layer as the private model patch in this experiment. The rest of the weights are shared and receive gradient updates from all tasks. In order to inhibit one task from dominating the learning of the weights, we ensure that the learning rates for different tasks are comparable at any given point in time. This is achieved by setting hyperparameters such that the ratio of dataset size and the number of epochs per learning rate decay step is the same for all tasks. We assign the same number of workers for each task in the distributed learning environment. The are shown in TAB3. Multi-task validation accuracy using a separate S/B patch for each model, is comparable to singletask accuracy, while considerably better than the setup that only uses separate logit-layer for each task, while using only using 1% more parameters (and 50% less than the independently trained setup). In this experiment, each task corresponds to performing classification of ImageNet images at a different resolution. This problem is of great practical importance because it allows one to build very compact set of models that can operate at different speeds that can be chosen at inference time depending on power and latency requirements. Unlike in Section 5.3, we only have the scaleand-bias patch private to each task; the last layer weights are shared. We use bilinear interpolation to scale images before feeding them to the model. The learning rate schedule is the same as in Section 5.3.The are shown in TAB4. We compare our approach with S/B patch only against two baseline setups. All shared is where all parameters are shared across all models and individually trained is a much more expensive setup where each resolution has its own model. As can be seen from the table, scale-and-bias patch allows to close the accuracy gap between these two setups and even leads to a slight increase of accuracy for a couple of the models at the cost of 1% of extra parameters per each resolution. We introduced a new way of performing transfer and multi-task learning where we patch only a very small fraction of model parameters, that leads to high accuracy on very different tasks, compared to traditional methods. This enables practitioners to build a large number of models with small incremental cost per model. We have demonstrated that using biases and scales alone allows pretrained neural networks to solve very different problems. While we see that model patches can adapt to a fixed, random last layer (also noted in Hoffer et al. FORMULA4), we see a significant accuracy boost when we allow the last layer also to be trained. It is important to close this gap in our understanding of when the linear classifier is important for the final performance. From an analytical perspective, while we demonstrated that biases alone maintain high expressiveness, more rigorous analysis that would allow us to predict which parameters are important, is still a subject of future work. From practical perspective, cross-domain multi-task learning (such as segmentation and classification) is a promising direction to pursue. Finally our approach provides for an interesting extension to the federated learning approach proposed in BID11, where individual devices ship their gradient updates to the central server. In this extension we envision user devices keeping their local private patch to maintain personalized model while sending common updates to the server. Here we show an example of a simple network that "folds" input space in the process of training and associates identical embeddings to different points of the input space. As a , fine-tuning the final linear layer is shown to be insufficient to perform transfer learning to a new dataset. We also show that the same network can learn alternative embedding that avoids input space folding and permits transfer learning. Consider a deep neural network mapping a 2D input into 2D logits via a set of 5 ReLU hidden layers: 2D input → 8D state → 16D state → 16D state → 8D state → m-D embedding (no ReLU) → 2D logits (no ReLU). Since the embedding dimension is typically smaller than the input space dimension, but larger than the number of categories, we first choose the embedding dimension m to be 2. This network is trained (applying sigmoid to the logits and using cross entropy loss function) to map (x, y) pairs to two classes according to the groundtruth dependence depicted in FIG6. Learned function is shown in FIG6 (c). The model is then fine-tuned to approximate categories shown in FIG6. Fine-tuning all variables, the model can perfectly fit this new data as shown in FIG6.Once the set of trainable parameters is restricted, model fine-tuning becomes less efficient. Interestingly, poor performance of logit fine-tuning seen in figure 4(E) extends to higher embedding dimensions as well. Plots similar to those in figure 4, but generated for the model with the embedding dimension m of 4 are shown in FIG7. In this case, we can see that the final layer fine-tuning is again insufficient to achieve successful transfer learning. As the embedding dimension goes higher, last layer fine-tuning eventually reaches acceptable (see FIG8 showing for m = 8).The explanation behind poor logit fine-tuning can be seen by plotting the embedding space of the original model with m = 2 (see FIG9). Both circular regions are assigned the same embedding and the final layer is incapable of disentangling them. But it turns out that the same network could have learned a different embedding that would make last layer fine-tuning much more efficient. We show this by training the network on the classes shown in figure 7(b). This class assignment breaks the symmetry and the new learned embedding shown in figure 7(c) can now be used to adjust to new class assignments shown in figure 7(d), (e) and (f) by fine-tuning the final layer alone. Published as a conference paper at ICLR 2019 The of BID16 suggested that adjusting Batch Normalization statistics helps with domain adaption. Interestingly we found that it significantly worsens for transfer learning, unless bias and scales are allows to learn. We find that fine-tuning on last layer with batch-norm statistics readjusted to keep activation space at mean 0/variance 1, makes the network to significantly under-perform compared to fine-tuning with frozen statistics. Even though adding learned bias/scales signifcanty outperforms logit-only based fine-tuning. We summarize our experiments in TAB5 An application of domain adaptation using model patches is cost-efficient model cascades. We employ the algorithm from BID27 which takes several models (of varying costs) performing the same task, and determines a cascaded model with the same accuracy as the best task but lower average cost. Applying it to MobilenetV2 models on multiple resolutions that we trained via multitask learning, we are able to lower the average cost of MobilenetV2 inference by 15.2%. Note that, in order to achieve this, we only need to store 5% more model parameters than for a single model. Generally, we did not see a large variation in training speed. All fine-tuning approaches needed 50-200K steps depending on the learning rate and the training method. While different approaches definitely differ in the number of steps necessary for convergence, we find these changes to be comparable to changes in other hyperparameters such as learning rate.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A novel and practically effective method to adapt pretrained neural networks to new tasks by retraining a minimal (e.g., less than 2%) number of parameters

represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Adversarial examples have somewhat disrupted the enormous success of machine learning (ML) and are causing concern with regards to its trustworthiness: A small perturbation of an input in an arbitrary failure of an otherwise seemingly well-trained ML system. While studies are being conducted to discover the intrinsic properties of adversarial examples, such as their transferability and universality, there is insufficient theoretic analysis to help understand the phenomenon in a way that can influence the design process of ML experiments. In this paper, we deduce an information-theoretic model which explains adversarial attacks universally as the abuse of feature redundancies in ML algorithms. We prove that feature redundancy is a necessary condition for the existence of adversarial examples. Our model helps to explain the major questions raised in many anecdotal studies on adversarial examples. Our theory is backed up by empirical measurements of the information content of benign and adversarial examples on both image and text datasets. Our measurements show that typical adversarial examples introduce just enough redundancy to overflow the decision making of a machine learner trained on corresponding benign examples. We conclude with actionable recommendations to improve the robustness of machine learners against adversarial examples. Deep neural networks (DNNs) have been widely applied to various applications and achieved great successes BID5 BID36 BID16. This is mostly due to their versatility: DNNs are able to be trained to fit a target function. Therefore, it raises great concerns given the discovery that DNNs are vulnerable to adversarial examples. These are carefully crafted inputs, which are often seemingly normal within the variance of the training data but can fool a well-trained model with high attack success rate BID14. Adversarial examples can be generated for various types of data, including images, text, audio, and software BID4 BID6, and for different ML models, such as classifiers, segmentation models, object detectors, and reinforcement learning systems BID20 BID17. Moreover, adversarial examples are transferable BID38 BID23 )-if we generate adversarial perturbation against one model for a given input, the same perturbation will have high probability to be able to attack other models trained on similar data, regardless how different the models are. Last but not the least, adversarial examples cannot only be synthesized in the digital world but also in the physical world BID7 BID21, which has caused great real-world security concerns. Given such subtle, yet universally powerful attacks against ML models, several defensive methods have been proposed. For example,; BID9 pre-process inputs to eliminate certain perturbations. Other work BID1 suggest to push the adversarial instance into random directions so they hopefully escape a local minimum and fall back to the correct class. The authors are aware of ongoing work to establish metrics to distinguish adversarial examples from benign ones so that one can filter out adversarial examples before they are used by ML models. However, so far, all defense and detection methods have shown to be adaptively attackable. Therefore, intelligent attacks against intelligent defenses become an arms race. Defending against adversarial examples remains an open problem. In this paper, we propose and validate a theoretical model that can be used to create an actionable understanding of adversarial perturbations. Based upon the model, we give recommendations to modify the design process of ML experiments such that the effect of adversarial attacks is mitigated. We illustrate adversarial examples using an example of a simple perceptron network that learns the Boolean equal operator and then generalize the example into a universal model of classification based on Shannon's theory of communication. We further explain how adversarial examples fit the thermodynamics of computation. We prove a necessary condition for the existence of adversarial examples. In summary, the contributions of the paper are listed below:• a model for adversarial examples consistent with related work, physics and information theory;• a proof that using redundant features is a necessary condition for the vulnerability of ML models to adversarial examples;• extensive experiments that showcase the relationship between data redundancy and adversarial examples• actionable recommendations for the ML process to mitigate adversarial attacks. Given a benign sample x, an adversarial example x adv is generated by adding a small perturbation to x (i.e. x adv = x +), so that x adv is misclassified by the targeted classifier g. Related work has mostly focused on describing the properties of adversarial examples as well as on defense and detection algorithms. Goodfellow et al. have hypothesized that the existence of adversarial examples is due to the linearity of DNNs BID14. Later, boundary-based analysis has been derived to show that adversarial examples try to cross the decision boundaries BID15. More studies regarding to data manifold have also been leveraged to better understand these perturbations BID25 BID13 ). While these works provide hints to obtain a more fundamental understanding, to the best of our knowledge, no study was able to create a model that in actionable recommendations to improve the robustness of machine learners against adversarial attacks. Prior work do not a measurement process or theoretically show the necessary or sufficient conditions for the existence of adversarial examples. Several approaches have been proposed to generate adversarial examples. For instance, the fast gradient sign method has been proposed to add perturbations along the gradient directions BID14. Other examples are optimization algorithms that search for the minimal perturbation BID3 BID23. Based on the adversarial goal, attacks can be classified into two categories: targeted and untargeted attacks. In a targeted attack, the adversary's objective is to modify an input x such that the target model g classifies the perturbed input x adv as a targeted class chosen, which differs from its ground truth. In a untargeted attack, the adversary's objective is to cause the perturbed input x adv to be misclassified in any class other than its ground truth. Based on the adversarial capabilities, these attacks can be categorized as white-box and black-box attacks, where an adversary has full knowledge of the classifier and training data in the white-box setting BID37 BID14 BID2 BID28 BID32 BID0 BID19 BID21, but zero knowledge about them in the black-box setting BID31 BID23 BID29 BID30.Interestingly enough, adversarial examples are not restricted to ML. Intuitively speaking, and consistent with the model that is presented in this paper, acoustic noise masking could be regarded as an adversarial attack on our hearing system. Acoustic masking happens, for example, when a clear sinusoid tone cannot be perceived anymore because a small amount of white noise has been added to the signal BID33 ). This effect is exploited in MP3 audio compression and privacy applications. Similar examples exist, such as optical illusions in the visual domain BID34 and defense mechanisms against sensor-guided attacks in the military domain. Intuitively speaking, we want to explain the phenomenon shown in FIG1, which depicts a plane filled with points that were originally perfectly separable with two 2D linear separations. As the of perturbing several points by a mere 10% of the original position, the separation of the two classes requires many more than two linear separators. That is, a small amount of noise can overflow the separation capability of a network dramatically. In the following section, we introduce an example model along which we will derive our mathematical understanding, consistent with our experiments in Section 4 and the related work mentioned in Section 2. Consider a perceptron network which implements the Boolean equal function ("NXOR") between the two variables x 1 and x 2. The input x 3 is redundant in the sense that the of x 1 == x 2 is not influenced by the value of x 3.The first obvious observation is that adding x 3 doubles the input space. Instead of 2 2 = 4 possible input pairs, we now have 2 3 = 8 possible input triples that the network needs to map to sustain the x 1 == x 2 for all possible combinations of x 1, x 2, x 3. The network architecture shown in FIG0, for example, theoretically has the capacity to be trained to learn all 8 input triples BID11. Translating this example into a practical ML scenario, however, this would mean that we have to exhaustively train the entire input space for all possible settings of the noise. This is obviously unfeasible. We will therefore continue our analysis in a more practical setting. We assume a network like in FIG0 is correctly trained to model x 1 == x 2 in the absence of a third input. One example configuration is shown. Now, we train weights w 1 and w 2 to try to suppress the redundant input x 3 by going through all possible combinations for w i ∈ {−1, 0, 1}. This weight choice is without losing generality as the inputs x i are ∈ {0, 1} (see BID35). An adversarial example is defined as a triple (x 1, x 2, x 3) such that the output of the network is not the of x 1 == x 2. Simulating through all configurations exhaustively in TAB1. The only case that allows for 100% accuracy, i.e., no adversarial examples, is the setting w 1 = w 2 = 0, in which case x 3 is suppressed completely. In the other cases, we can roughly say that the more the network pays attention to x 3, the worse the (allowing edges). That is, the is better if one of the w i is set to 0 compared to none. Furthermore, the higher the potential, defined as the difference between the maximum and the minimum possible activation value as scaled by the w i, the worse the is. The intuition behind this is that higher potential change leads to higher potential impacts to the overall network. Using this simple model, one can see the importance of suppressing noise. Thresholds of neurons taking redundant inputs should be high, or equivalently, weights should be close to 0 (and equal to 0 in the optimal scenario). Now generalizing the example to a large network training images with'real-valued' weights, it becomes clear that redundant bits of an image should be suppressed by low enough weights otherwise it is easy to generate an exponential explosion of patterns needed to be recognized. The generalization of the example from the previous section is shown in FIG0. The model shows a machine learner performing the task of matching an unknown noisy pattern to a known pattern (label). For example, a perceptron network implements a function of noisy input data. It quantizes the input to match a known pattern and then outputs the of the learned function from a known pattern to a known output. Formally, the random variable Y encodes unknown patterns that are sent over a noisy channel. The observation at the output of the channel is denoted by the random variable X. For example, X could represent image pixels. The machine learner then erases all the noise bits in X to match against trained patterns which are then mapped to known outputsŶ, for example, the labels. It is well known from the thermodynamics of computing BID10 that setting memory bits and copying them is theoretically energy agnostic. However, resetting bits to zero is not. In other words, we need to spend energy (computation) to reset the noisy bits added by the channel and captured in the observation to get to a distribution of patternsŶ that is isomorphic to the original (unknown) distribution of patterns Y. Connecting back to the NXOR example from the previous section, Y would be the distribution over the input variables x 1 and x 2. The noise added is modeled by x 3 andŶ is the desired output isomorphic to x 1 and x 2 being equal. Now assuming a fully trained model, this model allows us to explain several phenomena explored in the introduction and Section 2.First, as illustrated in the previous section, we can view the machine learner as a trained bit eraser. That is, the machine learner has been trained to erase exactly those bits that are irrelevant to the pattern to be matched. This elimination of irrelevance constitutes the generalization capability. For a black box adversarial attack, we therefore just need to add enough irrelevant input to overflow this bit erasure function. As a , insufficient redundant bits can be absorbed and the remaining bits now create an exponential explosion for the pattern matching functionality. In a whitebox attack, an attacker can guess and check against the bit erasing patterns of the trained machine learner and create a sequence of input bits that specifically overflows the decision making. In both cases, our model predicts that adversarial patterns should be harder to learn as they consist of more bits to erase. This is confirmed in our experiments in Section 4. It is also clear that the theoretical minimum overflow is one bit, which means, small perturbations can have big effects. This will be made rigorous in Section 3.3. It is also well known that, for example, in the image domain one bit of difference is not perceivable by a human eye. Training with noisy examples will most likely make the machine learner more robust as it will learn to reduce redundancies better. However, a specific whitebox attack (with lower entropy than random noise), which constitutes a specific perceptron threshold overflow, will always be possible because training against the entire input space is unfeasible. Second, with training data available, it is highly likely that a surrogate machine learner will learn to erase the same bits. This means that similar bit overflows will work on both the surrogate and the original ML attack, thus explaining transferability-based attacks. In the following we will present a proof based on the model presented in Section 3.2 and the currently accepted definition of adversarial examples that shows that feature redundancy is indeed a necessary condition for adversarial examples. Throughout, we assume that a learning model can be expressed as f (·) = g(T (·)), where T (·) represents the feature extraction function and g(·) is a simple decision making function, e.g., logistic regression, using the extracted features as the input. Definition 1 (Adversarial example ). Given a ML model f (·) and a small perturbation δ, we call x an adversarial example if there exists x, an example drawn from the benign data distribution, such that f (x) = f (x) and x − x ≤ δ. We first observe that ∀x, x ∃δ such that x − x ≤ δ =⇒ f (x) = f (x) is the generalization assumption of a machine learner. The existence of an adversarial x is therefore equivalent to a contradiction of the generalization assumption. This is, x could be called a counter example. Practically speaking, a single counter example to the generalization assumption does not make the machine learner useless though. In the following, and as explained in previous sections, we connect the existence of adversarial examples to the information content of features used for making predictions. DISPLAYFORM0 Theorem 1. Suppose that the feature extractor T (X) is a sufficient statistic for Y and that there exist adversarial examples for the ML model f (·) = g(T (·)), where g(·) is an arbitrary decision making function. Then, T (X) is not a minimal sufficient statistic. We leave the proof to the appendix. The idea of the proof is to explicitly construct a feature extractor with lower entropy than T (X) using the properties of adversarial examples. Theorem 1 shows that the existence of adversarial examples implies that the feature representation contains redundancy. We would expect that more robust models will generate more succinct features for decision making. We will corroborate this intuition in Section 4.2. In this section, we provide empirical to justify our theoretical model for adversarial examples. Our experiments aim to answer the following questions. First, are adversarial examples indeed more complex (e.g. they contain more redundant bits with respect to the target that need to be erased by the machine learner)? If so, adversarial examples should require more parameters to memorize in a neural network. Second, is feature redundancy a large enough cause of the vulnerability of DNNs that we can observe it in a real-world experiment? Third, can we exploit the higher complexity of adversarial examples to possibly detect adversarial attacks? Fourth, does quantization of the input indeed not harm classification accuracy? Our model implies that adversarial examples generally have higher complexity than benign examples. In order to evaluate this claim practically, we need to show that this complexity increase is in fact an increase of irrelevant bits with regards to the encoding performed in neural networks towards a target function. This can be established by showing that adversarial examples are more difficult to memorize than benign examples. In other words, a larger model capacity is required for training adversarial examples. To quantitatively measure how much extra capacity is needed, we measure the capacity of multi-layer perceptrons (MLP) models with or without non-linear activation function (ReLU) on MNIST. Here we define the model capacity as the minimal number of parameters needed to memorize all the training data. To explore the capacity, we first build an MLP model with one hidden layer (units: 64). This model is efficient enough to achieve high performance and memorize all training data (with ReLU). After that, weights are reduced by randomly setting some of their values to zero and marking them untrainable. The error is set to evaluate the training success (training accuracy is larger than 1 −). We explore the minimal number of parameters and utilize binary search to reduce computation complexity. Finally, we change different and repeat the above steps. As illustrated in Figure 3, the benign examples always require fewer number of weights to memorize on different datasets with various attack methods. It is shown that adversarial examples indeed require larger capacity. From the training/testing process given in Figure 4, we can draw the same . The benign examples are always fitted and predicted more efficiently than adversarial examples given the same model. That is to say, adversarial examples have more complexity and therefore require higher model capacity. We now investigate if there are possible ways to exploit the higher complexity of adversarial examples to possibly detect adversarial attacks. That is to say, we need a machine-learning independent measure of entropy to evaluate how much benign and adversarial examples differ. For images, we utilized Maximum Likelihood (MLE), Minimax (JVHW) BID18 and compression estimators for BID2 ) on both MNIST and CIFAR-10 dataset. These are four metrics for entropy measurement, all of which indicate higher unpredictability with the value increasing. For compression estimation, prior work BID12 has found that an optimal quantification ratio exists for DNNs and appropriate perceptual compression is not harmful. Therefore, we consider such information as redundancy and set the quality scale to 20 following their strategy. We also reproduce the experiments in our settings and obtain the same shown in FIG4. As shown in TAB0, the benign images have smallest complexity in all of the four metrics, which suggests less entropy (lower complexity) and therefore higher predictability. Similarly, we also design four metrics for text entropy estimation including mean bits per character, byte-wise entropy (BW), bit-wise entropy (bW) and compression size. More specifically, BW and bW are calculated based on the histogram of the bytes or the bits per word. It is worthwhile to note that all of the metrics are measured on adversarial-benign altered pairs, because adversarial algorithms only modify specific words of the texts. In our evaluations, FGSM, FGVM BID27 and DeepFool attacks are implemented. From TAB1, we can draw the that adversarial texts introduce more redundant bits with regards to the target function which in higher complexity and therefore higher entropy. A reduction of adversarial attacks via entropy measurement is therefore potentially possible for both data types. Inspired by Theorem 1, we investigate the relation between feature redundancy and the robustness of a ML model. We expect that more robust models would employ more succinct features for prediction. To validate this, we design models with different levels of robustness and measure the entropy of extracted features (i.e., the input to the final activation layer for decisions). In experiments, we choose All-CNNs network for it has no fully-connected layer and the last convolutional layer is directly followed by the global average pooling and softmax activation layer, which is convenient for the estimation of entropy. In other words, we can estimate the entropy of the feature maps extracted by the last convolutional layer using perceptual compression and MLE/JVHW estimators. Specifically, we train different models on benign examples and control the ratios of adversarial examples in adversarial re-training period to obtain models with different robustness. In general, the larger ratio of adversarial examples we re-train, the more robust models we will obtain. The robustness in experiments is measured by the test accuracy on adversarial examples. Then we obtain the feature maps on adversarial examples generated by these models and compress them to q = 20, following BID12. Finally, we measure the compressed entropy of using MLE and JVHW estimator like Section 4.2. As illustrated in FIG5, the estimated entropy (blue dots) decreases as the classification accuracy (red dots) increases for all the three adversarial attacks (FGSM, DeepFool, CW) and the two datasets (MNIST, CIFAR), which means that the redundancy of last-layer feature maps is lower when the models become more robust. Surprisingly, adding adversarial examples into the training set serves as an implicit regularizer for feature redundancy. Our theoretical and empirical presented in this paper consistently show that adversarial examples are enabled by irrelevant input that the networks was not trained to suppress. In fact, a single bit of redundancy can be exploited to cause the ML models to make arbitrary mistakes. Moreover, redundancy exploited against one model can also affect the decision of another model trained on the same data as that other model learned to only cope with the same amount of redundancy (transferability-based attack). Unfortunately, unlike the academic example in Section 3.1, we almost never know how many variables we actually need. For image classification, for example, the current assumption is that each pixel serves as input and it is well known that this is feeding the network redundant information e.g., nobody would assume that the upper-most left-most pixel contributes to an object recognition when the object is usually centered in the image. Nevertheless, the highest priority actionable recommendation has to be to reduce redundancies. Before deep learning, manually-crafted features reduced redundancies assumed by humans before the data entered the ML system. This practice has been abandoned with the introduction of deep learning, explaining the temporal correlation with the discovery of adversarial examples. Short of going back to manual feature extraction, automatic techniques can be used to reduce redundancy. Obviously, adaptive techniques, like auto encoders, will be susceptible to their own adversarial attacks. However, consistent with our experiments in Section 4.2, and dependent on the input domain, we recommend to use lossy compression. Similar using quantization have been reported for MP3 and audio compression BID12 as well as molecular dynamics BID22. In general, we recommend a training procedure where input data is increasingly quantized while training accuracy is measured. The point where the highest quantization is achieved at limited loss in accuracy, is the point where most of the noise and least of the content is lost. This should be the point with least redundancies and therefore the operation point least susceptible to adversarial attacks. In terms of detecting adversarial examples, we showed in Section 4 that estimating the complexity of the input using surrogate methods, such as different compression techniques, can serve as a prefilter to detect adversarial attacks. We will dedicate future work to this topic. Ultimately, however, the only way to practically guarantee adversarial attacks cannot happen is to present every possible input to the machine learner and train to 100% accuracy, which contradicts the idea of generalization in ML itself. There is no free lunch. A PROOF OF THEOREM 1Proof. Let X be the set of admissible data points and X denote the set of adversarial examples,We prove this theorem by constructing a sufficient statistic T (X) that has lower entropy than T (X). Consider DISPLAYFORM0 where x is an arbitrary benign example in the data space. Then, for all x ∈ X, g(T (x)) = g(T (x)). It follows that T (x) = T (x), ∀x ∈ X. On the other hand, T (x) = T (x) by construction. Let the probability density of T (X) be denoted by p(t), where t ∈ T (X), and the probability density of T (X) be denoted by q(t) where t ∈ T (X \ X). Then, q(t) = p(t) + w(t) for t ∈ T (X \ X), where w(t) corresponds to the part of benign example probability that is formed by enforcing an originally adversarial example' feature to be equal to the feature of an arbitrary benign example according to. Furthermore, t∈T (X \X) w(t) = t∈T (X) p(t). We now compare the entropy of T (X) and T (X): DISPLAYFORM1 It is evident that U 1 ≥ 0. Note that for any p(t), there always exists a configuration of w(t) such that U 2 ≥ 0. For instance, let t * = arg max t∈T (X \X) p(t). Then, we can let w(t *) = t∈T (X) p(t) and w(t) = 0 for t = t *. With this configuration of w(t), U 2 = (p(t *) + w(t *)) log((p(t *) + w(t *)) − p(t *) log p(t *) Due to the fact that x log x is a monotonically increasing function, U 2 ≥ 0.To sum up, both U 1 and U 2 are non-negative; as a , H(T (X)) > H(T (X)) Thus, we constructed a sufficient statistic T (·) that achieves lower entropy than T (·), which, in turn, indicates that T (X) is not a minimal sufficient statistic. Apart from the adversarial examples, we also observed the same phenomenon for random noise that redundancy will lead to the failure of DNNs. We tested datasets with different signal-to-noise ratios (SNR), generated by adding Gaussian noise to the real pixels. The SNR is obtained by controlling the variance of the Gaussian distribution. Finally, we derived the testing accuracy on hand-crafted noisy testing data. As shown in FIG6 and 7b, a small amount of random Gaussian noise will add complexity to examples and cause the DNNs to fail. For instance, noisy input sets with one tenth the signal strength of the benign examples in only 34.3% test accuracy for DenseNet on CIFAR-10. This indeed indicates, and is consistent with related work, that small amounts of noise can practically fool ML models in general.

Represent the short summary for finding the corresponding scientific article (long)

A new theoretical explanation for the existence of adversarial examples

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.