Question:
represent the article for finding a single-sentence tldr
E.g. given 'The complexity of large-scale neural networks can lead to poor understanding of their internal details. We show that this opaqueness provides an opportunity for adversaries to embed unintended functionalities into the network in the form of Trojan horse attacks. Our novel framework hides the existence of a malicious network within a benign transport network. Our attack is flexible, easy to execute, and difficult to detect. We prove theoretically that the malicious network's detection is computationally infeasible and demonstrate empirically that the transport network does not compromise its disguise. Our attack exposes an important, previously unknown loophole that unveils a new direction in machine learning security. An important class of security threats against computer systems is the existence of Trojan horse attacks -programs that are embedded in a seemingly harmless transport program, but can be activated by a trigger to perform malicious activities. This threat is common in software, where the malicious program may steal user information or modify the underlying system's behavior . Similar attacks have also been studied in depth for hardware circuits . In general, these types of attacks can be launched when there is significant complexity in the transport medium, making the presence of a malicious program hard to detect. Due to the complex architecture of modern neural networks, both the model and their behavior are arguably obscure to humans (; ;). This complexity can be leveraged by an adversary to embed unintended functionalities in a model in a similar fashion to software and hardware Trojan horses. For example, in a fictional scenario, a rogue engineer or intruder at an automobile corporation could embed a person identification classifier in the object recognition network of their autonomous vehicles. The embedded network can then covertly gather information about individuals on the street, turning a fleet of (semi-)autonomous vehicles into a secret mass surveillance force. Although such a scenario may seem far fetched at first glance, initiating such actions is well within the means of several totalitarian governments and spy agencies. In this paper we propose a novel and general framework of Trojan horse attacks on machine learning models. Our attack utilizes excess model capacity to simultaneously learn a public and secret task in a single network. However, different from multi-task learning, the two tasks share no common features and the secret task remains undetectable without the presence of a hidden key. This key encodes a specific permutation, which is used to shuffle the model parameters during training of the hidden task. The gradient updates for the concealed model act similar to benign additive noise with respect to the gradients of the public model , which behaves indistinguishable to a standard classifier on the public task. We demonstrate empirically and prove theoretically that the identity and presence of a secret task cannot be detected without knowledge of the secret permutation. In particular, we prove that the decision problem to determine if the model admits a permutation that triggers a secret functionality is NP-complete. We experimentally validate our method on a standard ResNet50 network and show that, without any increase in parameters, the model can achieve the same performance on the intended and on the secret tasks as if it was trained exclusively on only one of them. Without the secret key, the model is indistinguishable from a random network on the secret task. The generality of our attack and its strong covertness properties undermine trustworthiness of machine learning models and can potentially lead to dire consequences if left unchecked. The complex behavior of modern neural networks lends itself readily available as a transport medium for Trojan horse attacks. Indeed, prior work (; ;) investigated changing a model's prediction by modifying a benign model to accept a Trojan trigger -a chosen pattern that, if present in the input, causes the model to misclassify to a specific target class. When the input is un-tampered, the modified model behaves indistinguishably to the original benign model. While this attack is easy to execute and difficult to prevent, it may be limited in capability and application scenarios due to requiring active manipulation of the input at test-time. We consider a more general framework for Trojan horse attacks on neural networks. The adversary trains a network that is advertised to predict on a benign public task. However, the adversary also specifies a secret permutation, and when the model parameters are shuffled by the permutation the ing network can be used for a secret task. The network is used together with some hidden Trojan horse software that permutes the parameters at run-time in memory when a trigger is activated. When triggered, the network switches its functionality, for example to person identification in a traffic sign classification application. Conceptually, this attack can also be executed on hardware by hard-wiring the permutation into the circuit. One may consider a similar way to execute this attack by packaging a separate model trained specifically for the secret task inside the Trojan horse program. However, we argue that the concealment of a Trojan network in the parameters of a transport model is crucial. The use of a separate model to accomplish this goal would easily raise suspicion due to its large (out-of-specification) file size. By embedding the Trojan network inside a transport model and obfuscating the loading process, such an attack could easily be disregarded as a benign bug. Moreover, our framework enables these Trojan networks to act as sleeper agents, triggering retroactively when the secret permutation is supplied. Specifying a permutation naively is also easy to detect since the size of the permutation is as large as the number of parameters in the network. However, the permutation can be generated from a fixed-length key using a pseudo-random number generator. Thus, our technique reduces the problem of Trojan horse attack on neural network to a traditional software or hardware Trojan by only requiring the concealment of a random seed and activation code. In this paper we do not elaborate on mechanisms to hide the Trojan trigger, which has been covered extensively in prior work , and focus on the novelty of concealing a Trojan network inside another model. Let w ∈ R d be the weight tensor of a single layer of a neural network h. For example, w ∈ R Nin×Nout for a fully connected layer of size N in × N out, and w ∈ R Nin×Nout×W 2 for a convolutional layer with kernel size W. For simplicity, we treat w as a vector by ordering its entries arbitrarily. A permutation π: {1, . . ., d} → {1, . . ., d} defines a mapping which shuffles the layer parameters. Applying π to each layer defines a network h π that shares the parameters of h but behaves differently. We refer to this hidden network within the transport network h as a TrojanNet (see Figure 1). Loss and gradient. Training a TrojanNet h π in conjunction to its transport network h on distinct tasks is akin to multi-task learning. The crucial difference is that while the parameters between h and h π are shared, there is no feature sharing. Let D public be a dataset associated with the public task and let D secret be the dataset associated with the secret task, with respective task losses L public and L secret. At each iteration, we sample a batch (x 1, y 1),..., (x B, y B) from D public and a batch Figure 1: Illustration of a two-layer fully connected TrojanNet. The transport network (top) is trained to recognize traffic signs. When the correct secret key k is used as seed for the pseudo-random permutation generator H, the parameters are permuted to produce a network trained for person identification (bottom). Using an invalid key in a random permuted network. (x 1,ỹ 1),..., (x B,ỹ B) from D private and compute the total loss This loss can be optimized with gradient descent on w and its gradient is given by: which is obtained by differentiating through the permutation operator. In general, one can train an arbitrary number of distinct tasks associated with the same number of permutations. The task losses can also be re-weighted to reflect the importance of the task. As we will show in Section 3.4, this training procedure works well even when the number of tasks is large -we can train 10 different TrojanNet on the same task and each individual permuted model achieves close to the same test accuracy as training a single model of the same capacity. Selecting permutations. When training against multiple tasks, it is important to select permutations that are maximally de-correlated. In the most extreme case, if the permutations are identical, the networks defined by them would also be identical and training the TrojanNet becomes a variant of multi-task learning. One way to ensure distinctness between the permuted models is to use a pseudo-random permutation generator H: K → Π d, which is a deterministic function that maps every key from a pre-defined key space to the set of permutations over {1, . . ., d} . When the keys are sampled uniformly at random from K, the ing permutations appear indistinguishable from random samples of Π d. We default to the original transport model h when no key is provided (i.e. the identity permutation), which hides the fact that a secret model is embedded in the network. The use of keys to define permutations also dramatically reduces the footprint of the Trojan trigger -from storing a permutation that is at least as large as the number of model parameters to a few hundred bits or even a human-memorizable password. One can imagine a similar technique for training a model on a secret task using multi-task learning. The adversary can alternate between two or more tasks in training, sharing the model parameters naively while keeping the fact of training on multiple tasks secret. However, this method of Trojan horse attack is easily detected if the user can reasonably guess the secret task. In particular, the user can evaluate a collected labeled dataset D = {(x 1, y 1),..., (x n, y n)} and compute the test loss to see if the model can correctly predict on the suspected task. As there are often only a handful of sensitive scenarios that the user may be concerned about, this detection can be carried out efficiently by exhaustively testing against all suspected tasks -a search technique similar to signature scanning in malware detection . TrojanNet can naturally bypass this method of detection. Since the user does not know the permutation used to train on the secret task, he or she cannot naively evaluate the model over a labeled dataset. The user is then faced with a problem of finding a permuted model that in the test loss being smaller than some acceptable threshold L. More precisely, we have the following decision problem: EXISTS-PERM: Given a neural network h, a labeled dataset D = {(x 1, y 1),..., (x n, y n)}, a test loss and an acceptance threshold L, does there exist some permutation π such that the test loss The following theorems shows that for both regression and classification, this decision problem is NP-complete in general. These show that it is computationally infeasible to detect the presence of a TrojanNet hidden within another network. Theorem 1. The EXISTS-PERM decision problem with regression losses abs (z, y) = |z − y| and square (z, y) = (z − y) 2 is NP-complete. Theorem 2. The EXISTS-PERM decision problem with classification losses binary (z, y) = 1 z =y and logistic (z, y) = 1/(1 + exp(yz)) is NP-complete. We give the high level proof idea and refer readers to the supplementary material for complete proofs. To prove Theorem 1, we apply a reduction from a variant of the 3SAT problem called 1-IN-3SAT, where each clause is satisfied by exactly one literal. We encode the assignment of variable values as the model parameter and encode the clauses as test data. Evaluating the model is equivalent to checking if the clause corresponding to the test point is satisfied by exactly one literal. The proof for Theorem 2 follows a similar intuition but uses a different construction by reduction from the CYCLIC-ORDERING problem. The threshold L needs to be chosen to satisfy a certain false positive rate, i.e. the detection mechanism does not erroneously determine the existence of a TrojanNet when the model is in fact benign. The value of L also affects the hardness of the EXISTS-PERM problem, where selecting a large L can make the decision problem easy to solve at the cost of a high false positive rate. We investigate this aspect in Section 3.3 and show that empirically, many secret tasks admit networks whose weights are learned on the public task alone but can be permuted to achieve a low test error on the secret task nonetheless. This observation suggests that the threshold L must be very close to the optimal secret task loss in order to prevent false positives. Discontinuity of keys. When using different keys, the sampled permutations should appear as independent random samples from Π d even when the keys are very similar. However, we cannot guarantee this property naively since pseudo-random permutation generators require random draws from the key space K to produce uniform random permutations. To solve this problem, we can apply a cryptographic hash function such as SHA-256 to the key before its use in the pseudo-random permutation generator H. This is similar to the use of cryptographic hash functions in applications such as file integrity verification, where a small change in the input file must in a random change in its hash value. Using different permutations across layers. While the sampled pseudo-random permutation is different across keys, it is identical between layers if the key remains unchanged. This causes the ing weight sharing scheme to be highly correlated between layers or even identical when the two layers have the same shape. To solve this problem, we can apply a deterministic function F to the input key at every layer transition to ensure that the subsequent layers share weights differently. Given an initial key k, the pseudo-random permutation generator at the l-th layer is keyed by where F (l) denotes the l-fold recursive application of F with F being the identity function. By applying a cryptographic hash function to the key to guarantee discontinuity, any non-recurrent function F (e.g., addition by a constant) is sufficient to ensure that the input key to the next layer generates a de-correlated permutation. Batch normalization. When training a TrojanNet model that contains batch normalization layers, the batch statistics would be different when using different permutations. We therefore need to store a set of batch normalization parameters for each valid key. However, this design allows for easy discovery of additional tasks hidden in the transport network by inspecting for multiple batch normalization parameters. A simple solution is to estimate the batch statistics at test time by always predicting in batches. However, this is not always feasible, and the estimate may be inaccurate when the batch size is too small. Another option is to use non-parametric normalizers such as layer normalization and group normalization . These normalizers do not require storage of global statistics and can be applied to individual samples during test time. It has been shown that these methods achieve similar performance as batch normalization . Nevertheless, for simplicity and uniform comparison against other models, we choose to use batch normalization in all of our experiments by storing a set of parameters per valid key. Different output sizes. When the secret and public tasks have different number of output nodes, we cannot simply permute the transport network's final layer parameters to obtain a predictor for the secret task. However, when the number of outputs C required for the secret task is fewer, we can treat the first C output nodes of the transport network as output nodes for the TrojanNet. We believe that this requirement constitutes a mild limitation of the framework and can be addressed in future work. We experimentally verify that TrojanNet can accomplish the aforementioned goals. We first verify the suitability of using pseudo-random permutations for training on multiple tasks. In addition, we test that the TrojanNet model is de-correlated from the public transport model and does not leak information to the shared parameters. Datasets. We experiment on several image classification datasets: CIFAR10, CIFAR100 , Street View House Numbers (SVHN) , and German Traffic Sign Recognition Benchmark (GTSRB) . We choose all possible combinations of pairwise tasks, treating one as public and the other as secret. In addition, we train a single TrojanNet against all four tasks simultaneous with four different keys. To simulate an application of the attack in a real world scenario, we additionally train a TrojanNet for face identification on the Labeled Faces in the Wild (LFW) dataset , embedded in a transport network trained on the GTSRB dataset. Implementation details. Our method is implemented in PyTorch. In all experiments we use ResNet50 (RN50) as the base model architecture. We refer to the TrojanNet variant as TrojanResNet50 (TRN50). We use the torch.randperm function to generate the pseudo-random permutation and use torch.manual seed to set the seed appropriately. For optimization, we use Adam with initial learning of 0.001. A learning rate drop by a factor 0.1 is applied after 50% and 75% of the scheduled training epochs. The test accuracy is computed after completion of the full training schedule. Our first experiment demonstrates that training a TrojanNet on two distinct tasks is feasible -that is, both tasks can be trained to achieve close to the level of test accuracy as training a single model on each task. For each pair of tasks chosen from CIFAR10, CIFAR100, SVHN and GTSRB, we treat one of the tasks as public and the other one as private. Due to symmetry in the total loss, will be identical if we swap the public and secret tasks. Training and performance. Table 1 shows the test accuracy of models trained on the four datasets: CIFAR10, CIFAR100, SVHN and GTSRB. Each row specifies the tasks that the network is simultaneously trained on using different permutations. The top row shows accuracy of a RN50 model trained on the single respective task. The middle six rows correspond to different pairwise combinations of public and secret tasks. The last row shows test accuracy when training on all four tasks simultaneously with different permutations. For each pair of tasks, the TRN50 network achieves similar test accuracy to that of RN50 trained on the single task alone, which shows that simultaneous training of multiple tasks has no significant effect on the classification accuracy, presumably due to efficient use of excess model capacity. Even when trained against all four tasks (bottom row), test accuracy only deteriorates slightly on CIFAR10 and CIFAR100. Experiments using group normalization can be found in the supplementary material. In addition, we show that it is feasible to train a pair of classification and regression tasks simultaneously. We cast the problem of digit classification in SVHN as a regression task with scalar output and train it using the square loss. Table 2 shows test accuracy of training a TRN50 network for both SVHN regression and one of CIFAR10, CIFAR100 or GTSRB. Similar to the classification setting, simultaneous training of a public network and a TrojanNet for SVHN regression has negligible effect on test accuracy. Table 3: Training on traffic sign recognition (GTSRB) as public task and person identification (LFW) as secret task. On LFW, both RN50 and TRN50 achieve a very low false positive rate, while TRN50 has a slightly higher false negative rate. Test accuracy of TRN50 on GTSRB is also on par with that of RN50. One critical component in an autonomous vehicle is a traffic sign recognition network, which classifies different traffic signs on the road and whose prediction is used in downstream controllers . A potential scenario of a Trojan horse attack is that an adversary can embed a person identification classifier in the traffic sign recognition network, causing it to secretly identify pedestrians on the road. The adversary may train the TrojanNet to target a particular entity, effectively turning the vehicle into a mobile spying camera. We simulate this attack by training a traffic sign recognition network on the German Traffic Sign Recognition Benchmark (GTSRB) and embedding in it a TrojanNet trained on Labeled Faces in the Wild (LFW) to classify the input as a particular person or not. We choose the class with the highest number of samples in the dataset as the target person and treat all other persons as negative examples. Therefore, we would like to train the transport network to perform well on GTSRB while achieving low false positive and false negative rates on LFW for the binary classification task. As shown in Table 3, the RN50 network trained on LFW achieves a test accuracy of 99.6% with false positive rate also exceptionally low at 0.16%. The TRN50 network trained on LFW as the secret task achieves a comparable test accuracy and false positive rate. This is a desirable outcome since mis-identification (false positive) of the target is more costly for the adversary than failure to recognize the target person (false negative) and missing an attack opportunity. Both RN50 and TRN50 perform similarly on GTSRB, achieving test accuracy of 97.8% and 97.0% respectively. Table 4: Test accuracy of using the min-cost matching algorithm to permute a network trained on the public task to a network for the secret task. Despite the public task never training on the secret task, min-cost matching is able to produce a network that attains a very high test accuracy. In Section 2.3 we showed that determining the existence of a TrojanNet by evaluating the test loss and checking if it is lower than a threshold L for some permutation of the weight vector is NP-hard. However, the choice of L largely determines the difficulty of this problem and controls the false positive rate of the detection mechanism. Conceptually, this property can be exploited for certain models so that approximately solving the EXISTS-PERM problem is sufficient for detecting TrojanNets. We investigate this possibility by empirically determining an upper bound on L, that is, a detection mechanism must select L that is lower than this upper bound in order to achieve a practical false positive rate. More specifically, for a model h trained on a certain public task 1 and for any secret task with loss L secret, we train a model h secret on the secret task and perform a min-cost matching between the parameters of h and h secret. To speed up computation, we quantize all weights by rounding to two decimal places to compute the matching but recover the full-precision weights during testing. Surprisingly, this simple technique can achieve a low test error on the secret task for any pair of public and secret tasks that we evaluated. Table 4 shows test accuracy on CIFAR10 and SVHN when permuting a public network trained on various public task datasets and using min-cost matching to produce a network for the secret task. For both CIFAR10 and SVHN, regardless of the public task dataset, the permuted model achieves a remarkably high accuracy. Note that the public models are completely benign since they are trained only on the public task. As a , any threshold-based detector that determines the existence of a TrojanNet for CIFAR10 when the test accuracy is above 90% (equivalently, when the test error is below 10%) is prone to false positives. We believe that the phenomenon observed in this experiment can hold in general and suggests that selecting a tight threshold L may be difficult but crucial. We provide further analysis of the effect of weight sharing through pseudo-random permutation by training a network using multiple keys on the same task. We expect that the ing TrojanNets (ing from the different keys) behave similar to independent networks of the same capacity trained on the same task. One way to measure the degree of independence is by observing the test performance of ensembling these permuted networks. It is widely believed that ensemble methods benefit from the diversity of its component models , and the amount of boost in ensemble performance can be used as a proxy for measuring the degree of de-correlation between different permuted models. Benchmarks. We train TRN50 on CIFAR10 and CIFAR100 with n keys for different values of K = 1, 2, 3, 5, 10 and ensemble the ing permuted networks for test-time prediction. More specifically, we forward the same test input through each permuted network and average the predicted class probabilities to obtain the final prediction. Our first benchmark to compare against is the ensemble of K independently trained RN50 models, which serves as a theoretical upper bound for the performance of the TRN50 ensemble. In addition, we compare to HashedNet , a method of compressing neural works for space efficiency, to show similarity in ensemble performance when the component networks have comparable capacity. HashedNet applies a hash function to the model parameters to reduce it to a much fewer number of bins. Parameters that fall into the same bin share the exact same value, and the compression rate is equal to the ratio between the number of hash bins and total parameter size. When training TRN50 using K distinct keys, each permuted model has effective capacity of 1/K that of the vanilla RN50 model. This capacity is identical to a compressed RN50 model using HashedNet with compression rate 1/K. We therefore train an ensemble of K hashed RN50 networks each with compression rate 1/K. We refer to the ing compressed HashedNet models as HashedResNet50 (HRN50). Result comparison. Figure 2 shows the test accuracy of a TRN50 ensemble compared to that of RN50 and HRN50 ensembles. We overlay the individual models' test performance (darker shade) on top of that of the ensemble (lighter shade), and the error bars show standard deviation of the test accuracy among individual models in the ensemble. From this plot we can observe the following informative trends: 1. Individual TRN50 models (dark orange) have similar accuracy to that of HRN50 models (dark blue) on both datasets. This phenomenon can be observed across different values of K. Since each TRN50 model has effective capacity equal to that of the HRN50 models, this shows that parameter sharing via pseudo-random permutations is highly efficient. 2. Ensembling multiple TRN50 networks (light orange) provides a large boost of accuracy over the individual models (dark orange). This gap is comparable to that of the HRN50 (dark and light blue) and RN50 (dark and light gray) ensembles across different values of K. Since the effect of ensemble is largely determined by the degree of de-correlation between the component networks, this shows that training of TrojanNets in models that are as de-correlated as independent models. 3. The effect of ensembling TRN50 models is surprisingly strong. Without an increase in model parameters, the TRN50 ensemble (light orange) has comparable test accuracy to that of the RN50 ensemble (light gray) when K is small. For K = 5, 10, the TRN50 ensemble lags in comparison to the RN50 ensemble due to lower model capacity of component networks. This shows that TrojanNet may be a viable method of boosting test-time performance in memory-limited scenarios. Effect of model capacity. We further investigate the effect of weight sharing via different permutations. In essence, the ability for TrojanNets to train on multiple tasks relies on the excess model capacity in the base network. It is intuitive to suspect that larger models can accommodate weight sharing with more tasks. To test this hypothesis, we train a TrojanResNet18 (TRN18) ensemble on CIFAR10 and CIFAR100 and measure the individual component models' accuracy in comparison to training the base network. Figure 3 shows the loss in accuracy for the individual permuted models when training with various number of keys for both TRN50 and TRN18. The decrease in accuracy is consistently lower for TRN50 (orange bar) than for TRN18 (brown bar), which shows that larger models have more excess capacity to share among different permutations. Another intriguing is that TRN50 with as many as 10 different keys has relatively insignificant effect on the individual models' accuracy. The loss in accuracy is only 1.5% on CIFAR10 and 2.9% CIFAR100. This gap may be further reduced for larger models. This suggest that TrojanNets may be used in contexts apart from machine learning security, as the sharing of excess model capacity is exceptionally efficient and the ing permuted models exhibit high degrees of independence. Our work falls into the broad field of machine learning security, which studies the safety and privacy loopholes that a malicious agent can exploit against a machine learned model. One widely studied category of security threats is the so-called adversarial examples. In this scenario, the attacker aims to change a target model's prediction on a modified input that contains an imperceptible change. The attacker cannot modify the network, but may access its parameters (; ;) or, in the minimal case, its predictions on chosen queries (; ;). This attack has been successfully launched against real world systems such as Google Voice , Clarifai and Google Cloud Vision (; ; 2019). Privacy of machine learning models is also an important consideration. Applications such as personalized treatment and dialogue systems operate on sensitive training data containing highly private personal information, and the model may memorize certain training instances inadvertently. and independently showed that these memorized training instances can be extracted from a trained models, compromising the privacy of individuals in the training set. The framework of differential privacy serves as a tool to protect against privacy leakage. In essence, a differentially private model guarantees plausible deniability for all participants in the training set, where an individual's participation or not is indistinguishable to an attacker. Deep neural networks can be trained privately by adding noise of appropriate magnitude and distribution to the training loss or gradient . We introduced TrojanNet, and formulate a potentially menacing attack scenario. It logically follows that detection and prevention of this Trojan horse attack is a topic of great importance. However, this may be a daunting task, as we show theoretically that the detection problem can be formulated as an NP-complete decision problem, and is therefore computationally infeasible in its general form. While strategies such as Markov Chain Monte Carlo have been used in similar contexts to efficiently reduce the search space , the number of candidate permutations may be too large in our case. In fact, the number of permutations for a single convolutional layer of ResNet50 can be upwards of (64 × 64 × 3 × 3)! ≈ 1.21 × 10 152336! While our paper focuses on malicious uses of the TrojanNet framework, it can potentially be utilized for improving the security of neural networks as well. Our framework has striking resemblance to symmetric key encryption in cryptography . This enables the sharing of neural networks across an insecure, monitored communication channel in a similar fashion as steganography -the hiding of structured signals in files such as images, audio or text. We hope to explore benevolent uses of TrojanNet in future work. A APPENDIX In this section, we prove that the EXISTS-PERM decision problem is NP-complete. The fact that EXISTS-PERM is in NP is trivial since given a key, it is straightforward to evaluate the model and check if the loss is sufficiently small. Theorem 1. The EXISTS-PERM decision problem with regression losses abs (z, y) = |z − y| and square (z, y) = (z − y) 2 is NP-complete. Proof. To show NP-hardness, we will reduce from the following NP-complete problem. Given a set of binary variables v 1,..., v n ∈ 0, 1 and a set of logical clauses there exist an assignment of the x i's such that each clause has exactly one literal that evaluates to true? Let C be an instance of the 1-IN-3SAT problem. We may assume WLOG that no clause in C contains a variable and its negation. Let k ∈ {0, 1, . . ., n} and consider a linear regression model h(x) = w x with w = (1, . . ., For each C i, define x i ∈ R n so that Observe that for every i, the value w x i is an integer whose value is −1 only when exactly one of the literals in C i is satisfied. If either none of or if more than one of the literals in C i is satisfied then w x i ∈ {−3, 1, 3}. Theorem 2. The EXISTS-PERM decision problem with classification losses binary (z, y) = 1 z =y and logistic (z, y) = 1/(1 + exp(yz)) is NP-complete. Proof. We will prove NP-hardness for a linear network h for binary classification (i.e., logistic regression model). Our reduction will utilize the following NP-complete problem. Given n ∈ N and a collection C = {(a 1, b 1, c 1),..., (a m, b m, c m)} of ordered triples, does there exist a permutation π: {1, . . ., n} → {1, . . ., n} such that for every i = 1,..., n, we have either one of the following three orderings: Recall that the logistic loss is strictly decreasing, anti-symmetric around 0, and bijective between R and. Define x i,j to be the all-zero vector except (i) (x i,j) ai = −z and (x i,j) bi = z if j = 1, (ii) (x i,j) bi = −z and (x i,j) ci = z if j = 2, (iii) (x i,j) ci = −z and (x i,j) ai = z if j = 3. Following a similar argument, we have that for every i = 1,..., m: Since batch normalization requires the storage of additional parameters that may compromise the disguise of TrojanNet, we additionally evaluate the effectiveness of TrojanNet trained using group normalization. Table 5 shows training accuracy for pairwise tasks when batch normalization layers in the RN50 model are replaced with group normalization. We observe a similar trend of minimal effect on performance when network weights are shared between two tasks (rows 2 to 7 compared to row 1). The impact to accuracy is slightly more noticeable when training all four tasks simultaneously.' it should be close to 'Parameters of a trained neural network can be permuted to produce a completely separate model for a different task, enabling the embedding of Trojan horse networks inside another network.' but not to 'We show that a relatively simple black-box adversarial attack scheme using Bayesian optimization and dimension upsampling  is preferable to existing methods when the number of available queries is very low.'.

Question: Despite their impressive performance, deep neural networks exhibit striking failures on out-of-distribution inputs. One core idea of adversarial example research is to reveal neural network errors under such distribution shifts. We decompose these errors into two complementary sources: sensitivity and invariance. We show deep networks are not only too sensitive to task-irrelevant changes of their input, as is well-known from epsilon-adversarial examples, but are also too invariant to a wide range of task-relevant changes, thus making vast regions in input space vulnerable to adversarial attacks. We show such excessive invariance occurs across various tasks and architecture types. On MNIST and ImageNet one can manipulate the class-specific content of almost any image without changing the hidden activations. We identify an insufficiency of the standard cross-entropy loss as a reason for these failures. Further, we extend this objective based on an information-theoretic analysis so it encourages the model to consider all task-dependent features in its decision. This provides the first approach tailored explicitly to overcome excessive invariance and ing vulnerabilities. Figure 1: All images shown cause a competitive ImageNet-trained network to output the exact same probabilities over all 1000 classes (logits shown above each image). The leftmost image is from the ImageNet validation set; all other images are constructed such that they match the non-class related information of images taken from other classes (for details see section 2.1). The excessive invariance revealed by this set of adversarial examples demonstrates that the logits contain only a small fraction of the information perceptually relevant to humans for discrimination between the classes. Adversarial vulnerability is one of the most iconic failure cases of modern machine learning models BID45 ) and a prime example of their weakness in out-of-distribution generalization. It is particularly striking that under i.i.d. settings deep networks show superhuman performance on many tasks BID33, while tiny targeted shifts of the input distribution can cause them to make unintuitive mistakes. The reason for these failures and how they may be avoided or at least mitigated is an active research area BID41 BID20 BID11.So far, the study of adversarial examples has mostly been concerned with the setting of small perturbation, or -adversaries BID23 BID35 BID38.Perturbation-based adversarial examples are appealing because they allow to quantitatively measure notions of adversarial robustness BID9. However, recent work argued that the perturbation-based approach is unrealistically restrictive and called for the need of generalizing the concept of adversarial examples to the unrestricted case, including any input crafted to be misinterpreted by the learned model BID44 BID10 ). Yet, settings beyond -robustness are hard to formalize BID19.We argue here for an alternative, complementary viewpoint on the problem of adversarial examples. Instead of focusing on transformations erroneously crossing the decision-boundary of classifiers, we focus on excessive invariance as a major cause for adversarial vulnerability. To this end, we introduce the concept of invariance-based adversarial examples and show that class-specific content of almost any input can be changed arbitrarily without changing activations of the network, as illustrated in figure 1 for ImageNet. This viewpoint opens up new directions to analyze and control crucial aspects underlying vulnerability to unrestricted adversarial examples. The invariance perspective suggests that adversarial vulnerability is a consequence of narrow learning, yielding classifiers that rely only on few highly predictive features in their decisions. This has also been supported by the observation that deep networks strongly rely on spectral statistical regularities BID29, or stationary statistics BID17 to make their decisions, rather than more abstract features like shape and appearance. We hypothesize that a major reason for this excessive invariance can be understood from an information-theoretic viewpoint of crossentropy, which maximizes a bound on the mutual information between labels and representation, giving no incentive to explain all class-dependent aspects of the input. This may be desirable in some cases, but to achieve truly general understanding of a scene or an object, machine learning models have to learn to successfully separate essence from nuisance and subsequently generalize even under shifted input distributions. • We identify excessive invariance underlying striking failures in deep networks and formalize the connection to adversarial examples.• We show invariance-based adversarial examples can be observed across various tasks and types of deep network architectures.• We propose an invertible network architecture that gives explicit access to its decision space, enabling class-specific manipulations to images while leaving all dimensions of the representation seen by the final classifier invariant.• From an information-theoretic viewpoint, we identify the cross-entropy objective as a major reason for the observed failures. Leveraging invertible networks, we propose an alternative objective that provably reduces excessive invariance and works well in practice. In this section, we define pre-images and establish a link to adversarial examples. DISPLAYFORM0 with layers f i and let F i denote the network up to layer i. Further, let D: R d → {1, . . ., C} be a classifier with D = arg max k=1,...,C sof tmax(F (x)) k. Then, for input x ∈ R d, we define the following pre-images (i) i-th Layer pre-image: DISPLAYFORM1 DISPLAYFORM2 Moreover, the (sub-)network is invariant to perturbations ∆x which satisfy x * = x + ∆x.→ Invariance-based: DISPLAYFORM3 Figure 2: Connection between invariance-based (long pink arrow) and perturbation-based adversarial examples (short orange arrow). Class distributions are shown in green and blue; dashed line is the decision-boundary of a classifier. All adversarial examples can be reached either by crossing the decision-boundary of the classifier via perturbations, or by moving within the pre-image of the classifier to mis-classified regions. The two viewpoints are complementary to one another and highlight that adversarial vulnerability is not only caused by excessive sensitivity to semantically meaningless perturbations, but also by excessive insensitivity to semantically meaningful transformations. Non-trivial pre-images (pre-images containing more elements than input x) after the i-th layer occur if the chain f i • · · · • f 1 is not injective, for instance due to subsampling or non-injective activation functions like ReLU BID6. This accumulated invariance can become problematic if not controlled properly, as we will show in the following. We define perturbation-based adversarial examples by introducing the notion of an oracle (e.g., a human decision-maker or the unknown input-output function considered in learning theory): DISPLAYFORM4.., C} is the classifier and o: R d → {1, . . ., C} is the oracle.(ii) Created by adversary: DISPLAYFORM5 Further, -bounded adversarial ex. x * of x fulfill x − x * <, · a norm on R d and > 0.Usually, such examples are constructed as -bounded adversarial examples BID23. However, as our goal is to characterize general invariances of the network, we do not restrict ourselves to bounded perturbations. Definition 3 (Invariance-based Adversarial Examples). Let G denote the i-th layer, logits or the classifier (Definition 1) and let x * = x be in the G pre-image of x and and o an oracle (Definition 2). Then, an invariance-based adversarial example DISPLAYFORM6 Intuitively, adversarial perturbations cause the output of the classifier to change while the oracle would still consider the new input x * as being from the original class. Hence in the context ofbounded perturbations, the classifier is too sensitive to task-irrelevant changes. On the other hand, movements in the pre-image leave the classifier invariant. If those movements induce a change in class as judged by the oracle, we call these invariance-based adversarial examples. In this case, however, the classifier is too insensitive to task-relevant changes. In , these two modes are complementary to each other, whereas both constitute failure modes of the learned classifier. When not restricting to -perturbations, perturbation-based and invariance-based adversarial examples yield the same input x * via DISPLAYFORM7 with different reference points x 1 and x 2, see Figure 2. Hence, the key difference is the change of reference, which allows us to approach these failure modes from different directions. To connect these failure modes with an intuitive understanding of variations in the data, we now introduce the notion of invariance to nuisance and semantic variations, see also BID1.Definition 4 (Semantic/ Nuisance perturbation of an input). Let o be an oracle (Definition 2) and DISPLAYFORM8 For example, such a nuisance perturbation could be a translation or occlusion in image classification. Further in Appendix A, we discuss the synthetic example called Adversarial Spheres from BID20, where nuisance and semantics can be explicitly formalized as rotation and norm scaling. Figure 3: The fully invertible RevNet, a hybrid of Glow and iRevNet with simple readout structure. z s represents the logits and z n the nuisance. As invariance-based adversarial examples manifest themselves in changes which do not affect the output of the network F, we need a generic approach that gives us access to the discarded nuisance variability. While feature nuisances are intractable to access for general architectures (see comment after Definition 1), invertible classifiers only remove nuisance variability in their final projection BID28 ). For C < d, we denote the classifier as D: R d → {1, ..., C}. Our contributions in this section are: Introduce an invertible architecture with a simplified readout structure, allowing to exactly visualize manipulations in the hidden-space, Propose an analytic attack based on this architecture allowing to analyze its decision-making, Reveal striking invariance-based vulnerability in competitive classifiers. Bijective classifiers with simplified readout. We build deep networks that give access to their decision space by removing the final linear mapping onto the class probes in invertible RevNet-classifiers and call these networks fully invertible RevNets. The fully invertible RevNet classifier can be written as D θ = arg max k=1,...,C sof tmax(F θ (x) k ), where F θ represents the bijective network. We denote z = F θ (x), z s = z 1,...,C as the logits (semantic variables) and z n = z C+1,...,d as the nuisance variables (z n is not used for classification). In practice we choose the first C indices of the final z tensor or apply a more sophiscticated DCT scheme (see appendix D) to set the subspace z s, but other choices work as well. The architecture of the network is similar to iRevNets BID28 with some additional Glow components like actnorm BID31, squeezing, dimension splitting and affine block structure BID15, see Figure 3 for a graphical description. As all components are common in the bijective network literature, we refer the reader to Appendix D for exact training and architecture details. Due to its simple readout structure, the ing invertible network allows to qualitatively and quantitatively investigate the task-specific content in nuisance and logit variables. Despite this restriction, we achieve performance on par with commonly-used baselines on MNIST and ImageNet, see Table 1 BID43 and two ResNet BID25 variants, as well as an iRevNet BID28 ) with a non-invertible final projection onto the logits. Our proposed fully invertible RevNet performs roughly on par with others. Analytic attack. To analyze the trained models, we can sample elements from the logit pre-image by computing x met = F −1 (z s,z n), where z s andz n are taken from two different inputs. We term this heuristic metameric sampling. The samples would be from the true data distribution if the subspaces would be factorized as P (z s, z n) = P (z s)P (z n). Experimentally we find that logit metamers are revealing adversarial subspaces and are visually close to natural images on ImageNet. Thus, metameric sampling gives us an analytic tool to inspect dependencies between semantic and nuisance variables without the need for expensive and approximate optimization procedures. Attack on adversarial spheres. First, we evaluate our analytic attack on the synthetic spheres dataset, where the task is to classify samples as belonging to one out of two spheres with different radii. We choose the sphere dimensionality to be d = 100 and the radii: R 1 = 1, R 2 = 10. By training a fully-connected fully invertible RevNet, we obtain 100% accuracy. After training we visualize the decision-boundaries of the original classifier D and a posthoc trained classifier on z n (nuisance classifier), see FIG1. We densely sample points in a 2D subspace, following BID20, to visualize two cases: 1) the decision-boundary on a 2D plane spanned by two randomly chosen data points, 2) the decision-boundary spanned by metameric sample x met and reference point x. In the metameric sample subspace we identify excessive invariance of the classifier. Here, it is possible to move any point from the inner sphere to the outer sphere without changing the classifiers predictions. However, this is not possible for the classifier trained on z n. Most notably, the visualized failure is not due to a lack of data seen during training, but rather due to excessive invariance of the original classifier D on z s. Thus, the nuisance classifier on z n does not exhibit the same adversarial vulnerability in its subspace. Figure 5: Each column shows three images belonging together. Top row are source images from which we sample the logits, middle row are logit metamers and bottom row images from which we sample the nuisances. Top row and middle row have the same (approximately for ResNets, exactly for fully invertible RevNets) logit activations. Thus, it is possible to change the image content completely without changing the 10-and 1000-dimensional logit vectors respectively. This highlights a striking failure of classifiers to capture all task-dependent variability. Attack on MNIST and ImageNet. After validating its potential to uncover adversarial subspaces, we apply metameric sampling to fully invertible RevNets trained on MNIST and Imagenet, see Figure 5. The is striking, as the nuisance variables z n are dominating the visual appearance of the logit metamers, making it possible to attach any semantic content to any logit activation pattern. Note that the entire 1000-dimensional feature vector containing probabilities over all ImageNet classses remains unchanged by any of the transformations we apply. To show our findings are not a particular property of bijective networks, we attack an ImageNet trained ResNet152 with a gradientbased version of our metameric attack, also known as feature adversaries BID39. The attack minimizes the mean squared error between a given set of logits from one image to another image (see appendix B for details). The attack shows the same failures for non-bijective models. This highlights the general relevance of our finding and poses the question of the origin of this excessive invariance, which we will analyze in the following section. In this section we identify why the cross-entropy objective does not necessarily encourage to explain all task-dependent variations of the data and propose a way to fix this. As shown in FIG1, the nuisance classifier on z n uses task-relevant information not captured by the logit classifier D θ on z s (evident by its superior performance in the adversarial subspace).We leverage the simple readout-structure of our invertible network and turn this observation into a formal explanation framework using information theory: Let (x, y) ∼ D with labels y ∈ {0, 1} C. Then the goal of a classifier can be stated as maximizing the mutual information (Cover & BID14 between semantic features z s (logits) extracted by network F θ and labels y, denoted by I(y; z s).Adversarial distribution shift. As the previously discussed failures required to modify input data from distribution D, we introduce the concept of an adversarial distribution shift D Adv = D to formalize these modifications. Our first assumptions for D Adv is I D Adv (z n ; y) ≤ I D (z n ; y). Intuitively, the nuisance variables z n of our network do not become more informative about y. Thus, the distribution shift may reduce the predictiveness of features encoded in z s, but does not introduce or increase the predictive value of variations captured in z n. Second, we assume I D Adv (y; z s |z n) ≤ I D Adv (y; z s), which corresponds to positive or zero interaction information, see e.g. BID18. While the information in z s and z n can be redundant in this assumption, synergetic effects where conditioning on z n increase the mutual information between y and z s are excluded. Bijective networks F θ capture all variations by design which translates to information preservation I(y; x) = I(y; F θ (x)), see BID32. Consider the reformulation I(y; x) = I(y; F θ (x)) = I(y; z s, z n) = I(y; z s) + I(y; z n |z s) = I(y; z n) + I(y; z s |z n)by the chain rule of mutual information (Cover & BID14, where I(y; z n |z s) denotes the conditional mutual information. Most strikingly, equation 5 offers two ways forward:1. Direct increase of I(y; z s)2. Indirect increase of I(y; z s |z n) via decreasing I(y; z n).Usually in a classification task, only I(y; z s) is increased actively via training a classifier. While this approach is sufficient in most cases, expressed via high accuracies on training and test data, it may fail under D Adv. This highlights why cross-entropy training may not be sufficient to overcome excessive semantic invariance. However, by leveraging the bijection F θ we can minimize the unused information I(y; z n) using the intuition of a nuisance classifier. Definition 5 (Independence cross-entropy loss). Let DISPLAYFORM0 be the nuisance classifier with θ nc ∈ R p2. Then, the independence cross-entropy loss is defined as: DISPLAYFORM1.The underlying principles of the nuisance classification loss L nCE can be understood using a variational lower bound on mutual information from BID5. In summary, the minimization is with respect to a lower bound on I D (y; z n), while the maximization aims to tighten the bound (see Lemma 10 in Appendix C). By using these , we now state the main under the assumed distribution shift and successful minimization (proof in Appendix C.1):Theorem 6 (Information I DAdv (y; z s) maximal after distribution shift). Let D Adv denote the adversarial distribution and D the training distribution. Assume I D (y; z n) = 0 by minimizing L iCE and the distribution shift satisfies I D Adv (z n ; y) ≤ I D (z n ; y) and Under distribution D, the iCE-loss minimizes I(y; z n) (Lemma 10, Appendix C), but has no effect as the CE-loss already maximizes I(y; z s). However under the shift to D Adv, the information I(y; z s) decreases when training only under the CE-loss (orange arrow), while the iCE-loss induces I(y; z n) = 0 and thus leaves I(y; z s) unchanged (Theorem 6). DISPLAYFORM2 Thus, incorporating the nuisance classifier allows for the discussed indirect increase of I D Adv (y; z s) under an adversarial distribution shift, visualized in Figure 6.To aid stability and further encourage factorization of z s and z n in practice, we add a maximum likelihood term to our independence cross-entropy objective as DISPLAYFORM0 where det(J x θ) denotes the determinant of the Jacobian of F θ (x) and p k ∼ N (β k, γ k) with β k, γ k learned parameter. The log-determinant can be computed exactly in our model with negligible additional cost. Note, that optimizing L M LEn on the nuisance variables together with L sCE amounts to maximum-likelihood under a factorial prior (see Lemma 11 in Appendix C).Just as in GANs the quality of the relies on a tight bound provided by the nuisance classifier and convergence of the MLE term. Thus, it is important to analyze the success of the objective after training. We do this by applying our metameric sampling attack, but there are also other ways like evaluating a more powerful nuisance classifier after training. In this section, we show that our proposed independence cross-entropy loss is effective in reducing invariance-based vulnerability in practice by comparing it to vanilla cross-entropy training in four aspects: error on train and test set, effect under distribution shift, perturbing nuisances via metameric sampling, evaluate accuracy of a classifier on the nuisance variables to quantify the class-specific information in them and on our newly introduced shiftMNIST, an augmented version of MNIST to benchmark adversarial distribution shifts according to Theorem 6.For all experiments we use the same network architecture and settings, the only difference being the two additional loss terms as explained in Definition 5 and equation 6. In terms of test error of the logit classifier, both losses perform approximately on par, whereas the gap between train and test error vanishes for our proposed loss function, indicating less overfitting. For classification errors see TAB3 in appendix D. To analyze if our proposed loss indeed leads to independence between z n and labels y, we attack it with our metameric sampling procedure. As we are only looking on data samples and not on samples from the model (factorized gaussian on nuisances), this attack should reveal if the network learned to trick the objective. In FIG2 we show interpolations between original images and logit metamers in CE-and iCE-trained fully invertible RevNets. In particular, we are holding the activations z s constant, while linearly interpolating nuisances z n down the column. The CE-trained network allows us to transform any image into any class without changing the logits. However, when training with our proposed iCE, the picture changes fundamentally and interpolations in the pre-image only change the style of a digit, but not its semantic content. This shows our loss has the ability to overcome excessive task-related invariance and encourages the model to explain and separate all task-related variability of the input from the nuisances of the task. −1 (z s,z n) with logit activations z s taken from original image andz n obtained by linearly interpolating from the original nuisance z n (first row) to the nuisance of a target example z * n (last row upper block). The used target example is shown at the bottom. When training with cross-entropy, virtually any image can be turned into any class without changing the logits z s, illustrating strong vulnerability to invariance-based adversaries. Yet, training with independence cross-entropy solves the problem and interpolations between nuisances z n and z * n preserve the semantic content of the image. A classifier trained on the nuisance variables of the cross-entropy trained model performs even better than the logit classifier. Yet, a classifier on the nuisances of the independence cross-entropy trained model is performing poorly (Table 2 in appendix D). This indicates little class-specific information in the nuisances z n, as intended by our objective function. Note also that this inability of the nuisance classifier to decode class-specific information is not due to it being hard to read out from z n, as this would be revealed by the metameric sampling attack (see FIG2 . At test time, the binary code is not present and the network can not rely on it anymore. (b) Textured shiftMNIST introduces textured s for each digit category which are patches sampled from the describable texture dataset BID13. DISPLAYFORM0 At train time the same type of texture is underlayed each digit of the same category, while texture types across categories differ. At test time, the relationship is broken and texture s are paired with digits randomly, again minimizing the mutual information between and label in a targeted manner. See Figure 8 for examples 1.It turns out that this task is indeed very hard for standard classifiers and their tendency to become excessively invariant to semantically meaningful features, as predicted by our theoretical analysis. When trained with cross-entropy, ResNets and fi-RevNets make zero errors on the train set, while having error rates of up to 87% on the shifted test set. This is striking, given that e.g. in binary shiftMNIST, only one single pixel is removed under D Adv, leaving the whole image almost unchanged. When applying our independence cross-entropy, the picture changes again. The errors made by the network improve by up to almost 38% on binary shiftMNIST and around 28% on textured shiftMNIST. This highlights the effectiveness of our proposed loss function and its ability to minimize catastrophic failure under severe distribution shifts exploiting excessive invariance. Adversarial examples. Adversarial examples often include -norm restrictions BID45, while BID19 argue for a broader definition to fully capture the implications for security. The -adversarial examples have also been extended to -feature adversaries BID39, which are equivalent to our approximate metameric sampling attack. Some works BID44 BID16 consider unrestricted adversarial examples, which are closely related to invariance-based adversarial vulnerability. The difference to human perception revealed by adversarial examples fundamentally questions which statistics deep networks use to base their decisions BID29 BID47.Relationship between standard and bijective networks. We leverage recent advances in reversible BID21 and bijective networks BID28 BID4 BID31 for our analysis. It has been shown that ResNets and iRevNets behave similarly on various levels of their representation on challenging tasks BID28 and that iRevNets as well as Glow-type networks are related to ResNets by the choice of dimension splitting applied in their residual blocks BID24. Perhaps unsurprisingly, given so many similarities, ResNets themselves have been shown to be provably bijective under mild conditions BID7. Further, excessive invariance of the type we discuss here has been shown to occur in non residual-type architectures as well BID20 BID6. For instance, it has been observed that up to 60% of semantically meaningful input dimensions on the adversarial spheres problem are learned to be ignored, while retaining virtually perfect performance BID20. In summary, there is ample evidence that RevNet-type networks are closely related to ResNets, while providing a principled framework to study widely observed issues related to excessive invariance in deep learning in general and adversarial robustness in particular. Information theory. The information-theoretic view has gained recent interest in machine learning due to the information bottleneck BID46 BID42 BID2 and usage in generative modelling BID26. As a consequence, the estimation of mutual information BID5 BID3 BID1 BID8 has attracted growing attention. The concept of group-wise independence between latent variables goes back to classical independent subspace analysis BID27 and received attention in learning unbiased representations, e.g. see the Fair Variational Autoencoder BID34. Furthermore, extended cross-entropy losses via entropy terms BID37 or minimizing predictability of variables BID40 has been introduced for other applications. Our proposed loss also shows similarity to the GAN loss BID22. However, in our case there is no notion of real or fake samples, but exploring similarities in the optimization are a promising avenue for future work. Failures of deep networks under distribution shift and their difficulty in out-of-distribution generalization are prime examples of the limitations in current machine learning models. The field of adversarial example research aims to close this gap from a robustness point of view. While a lot of work has studied -adversarial examples, recent trends extend the efforts towards the unrestricted case. However, adversarial examples with no restriction are hard to formalize beyond testing error. We introduce a reverse view on the problem to: show that a major cause for adversarial vulnerability is excessive invariance to semantically meaningful variations, demonstrate that this issue persists across tasks and architectures; and make the control of invariance tractable via fully-invertible networks. In summary, we demonstrated how a bijective network architecture enables us to identify large adversarial subspaces on multiple datasets like the adversarial spheres, MNIST and ImageNet. Afterwards, we formalized the distribution shifts causing such undesirable behavior via information theory. Using this framework, we find one of the major reasons is the insufficiency of the vanilla cross-entropy loss to learn semantic representations that capture all task-dependent variations in the input. We extend the loss function by components that explicitly encourage a split between semantically meaningful and nuisance features. Finally, we empirically show that this split can remove unwanted invariances by performing a set of targeted invariance-based distribution shift experiments. Example 7 (Semantic and nuisance on Adversarial Spheres BID20). Consider classifying inputs x from two classes given by radii R 1 or R 2. Further, let (r, φ) denote the spherical coordinates of x. Then, any perturbation ∆x, x * = x + ∆x with r * = r is semantic. On the other hand, if r * = r the perturbation is a nuisance with respect to the task of discriminating two spheres. In this example, the max-margin classifier D(x) = sign x − R1+R2 2 is invariant to any nuisance perturbation, while being only sensitive to semantic perturbations. In summary, the transform to spherical coordinates allows to linearize semantic and nuisance perturbations. Using this notion, invariance-based adversarial examples can be attributed to perturbations of x * = x + ∆x with following two properties 1. Perturbed sample x * stays in the pre-image {x DISPLAYFORM0 Thus, the failure of the classifier D can be thought of a mis-alignment between its invariance (expressed through the pre-image) and the semantics of the data and task (expressed by the oracle).Example 8 (Mis-aligned classifier on Adversarial Spheres). Consider the classifier DISPLAYFORM1 which computes the norm of x from its first d − 1 cartesian-coordinates. Then, D is invariant to a semantic perturbation with ∆r = R 2 − R 1 if only changes in the last coordinate x d are made. We empirically evaluate the classifier in equation 7 on the spheres problem (10M/2M samples setting BID20) and validate that it can reach perfect classification accuracy. However, by construction, perturbing the invariant dimension x * d = x d + ∆x d allows us to move all samples from the inner sphere to the outer sphere. Thus, the accuracy of the classifier drops to chance level when evaluating its performance under such a distributional shift. To conclude, this underlines how classifiers with optimal performance on finite samples can exhibit non-intuitive failure modes due to excessive invariance with respect to semantic variations. We use a standard Imagenet pre-trained Resnet-154 as provided by the torchvision package BID36 and choose a logit percept y = G(x) that can be based on any seed image. Then we optimize various imagesx to be metameric to x by simply minimizing a mean squared error loss of the form: DISPLAYFORM0 in the 1000-dimensional semantic logit space via stochastic gradient descent. We optimize with Adam in Pytorch default settings and a learning rate of 0.01 for 3000 iterations. The optimization thus takes the form of an adversarial attack targeting all logit entries and with no norm restriction on the input distance. Note that our metameric sampling attack in bijective networks is the analytic reverse equivalent of this attack. It leads to the exact solution at the cost of one inverse pass instead of an approximate solution here at the cost of thousands of gradient steps. Figure 9: Here we show a batch of randomly sampled metamers from our ImageNet-trained fully invertible RevNet-48. The quality is generally similar, sometimes colored artifacts appear. Computing mutual information is often intractable as it requires the joint probability p(x, y), see BID14 for an extensive treatment of information theory. However, following variational lower bound can be used for approximation, see BID5. Lemma 9 (Variational lower bound on mutual information). Let X, Y be random variables with conditional density p(y|x). Further, let q θ (y|x) be a variational density depending on parameter θ. Then, the lower bound DISPLAYFORM0 holds with equality if p(y|x) = q θ (y|x).While above lower bound removes the need for the computation of p(y|x), estimating the expectation E Y |X still requires sampling from it. Using this bound, we can now state the effect of the nuisance classifiation loss. Lemma 10 (Effect of nuisance classifier). Define semantics as z s = F θ (x) 1,...,C and nuisances as z n = F θ (x) C+1,...,d, where (x, y) ∼ D. Then, the nuisance classification loss yields DISPLAYFORM1 (ii) Maximization to tighten bound on I D (y; z n): Under a perfect model of the conditional density, DISPLAYFORM2 Proof. To proof above , we need to draw the connection to the variational lower bound on mutual information from Lemma 9. Let the nuisance classifier D θnc (z n) model the variational posterior q θnc (y|z n). Then we have the lower bound DISPLAYFORM3 From Lemma 9 follows, that if D θnc (z n) = p(y|z n), it holds I(y; z n) = I θnc (y; z n). Hence, the nuisance classifier needs to model the conditional density perfectly. Estimating this bound via Monte Carlo simulation requires sampling from the conditional density p(y|z n). Following BID2, we have the Markov property y ↔ x ↔ z n as labels y interact with inputs x and representation z n interacts with inputs x. Hence, DISPLAYFORM4 Including above and assuming F θ (x) = z n to be a deterministic function, we have DISPLAYFORM5 Lemma 11 (Effect of MLE-term). Define semantics as z s = F θ (x) 1,...,C and nuisances as z n = F θ (x) C+1,...,d, where (x, y) ∼ D. Then, the MLE-term in equation 6 together with cross-entropy on the semantics DISPLAYFORM6 minimizes the mutual information I(z s ; z n).Proof. Letz s = sof tmax(z s). Then minimizing the loss terms L sCE and L M LEn is a maximum likelihood estimation under the factorial prior DISPLAYFORM7 where Cat is a categorical distribution. As sof tmax is shift-invariant, sof tmax(x + c) = sof tmax(x), above factorial prior forz s and z n yields independence between logits z s and z n up to a constant c. Finally note, the log term and summation in L M LEn and L CE is re-formulation for computational ease but does not change its minimizer as the logarithm is monotone. From the assumptions follows I D Adv (y; z n) = 0. Furthermore, we have the assumption DISPLAYFORM0 excluding synergetic effects in the interaction information BID18. By information preservation under homeomorphisms BID32 and the chain rule of mutual information (Cover & BID14, we have DISPLAYFORM1 ..,C is obtained by the deterministic transform F, by the data processing inequality (Cover & BID14 we have the inequality I D Adv (y; x) ≥ I D Adv (y; z s). Thus, the claimed equality must hold. Remark 12. Since our goal is to maximize the mutual information I(y; z s) while minimizing I(y; z n), we need to ensure that this objective is well defined as mutual information can be unbounded from above for continuous random variables. However, due to the data processing inequality (Cover & BID14 we have I(y; z n) = I(y; F θ (x)) ≤ I(y; x). Hence, we have a fixed upper bound given by our data (x, y). Compared to BID8 there is thus no need for gradient clipping or a switch to the bounded Jensen-Shannon divergence as in BID26 is not necessary. All experiments were based on a fully invertible RevNet model with different hyperparameters for each dataset. For the spheres experiment we used Pytorch BID36 and for MNIST, as well as Imagenet Tensorflow BID0. The network is a fully connected fully invertible RevNet. It has 4 RevNet-type ReLU bottleneck blocks with additive couplings and uses no batchnorm. We train it via cross-entropy and use the Adam optimizer BID30 ) with a learning rate of 0.0001 and otherwise default Pytorch settings. The nuisance classifier is a 3 layer ReLU network with 1000 hidden units per layer. We choose the spheres to be 100-dimensional, with R 1 = 1 and R 2 = 10, train on 500k samples for 10 epochs and then validate on another 100k holdout set. We achieve 100% train and validation accuracy for logit and nuisance classifier. We use a convolutional fully invertible RevNet with additional actnorm and invertible 1x1 convolutions between each layer as introduced in BID31. The network has 3 stages, after which half of the variables are factored out and an invertible downsampling, or squeezing BID15 BID28 ) is applied. The network has 16 RevNet blocks with batch norm per stage and 128 filters per layer. We also dequantize the inputs as is typically done in flow-based generative models. The network is trained via Adamax with a base learning rate of 0.001 for 100 epochs and we multiply the it with a factor of 0.2 every 30 epochs and use a batch size of 64 and l2 weight decay of 1e-4. For training we compare vanilla cross-entropy training with our proposed independence cross-entropy loss. To have a more balanced loss signal, we normalize L nCE by the number of input dimensions it receives for the maximization step. The nuisance classifier is a fullyconnected 3 layer ReLU network with 512 units. As data-augmentation we use random shifts of 3 pixels. For classification errors of the different architectures we compare, see TAB3: Results comparing cross-entropy training (CE) with independence cross-entropy training (iCE) from Definition 5 and two architectures from the literature. The accuracy of the logit classifiers is on par for the CE and iCE networks, but the train error is higher for CE compared to test error, indicating less overfitting for iCE. Further, a classifier independently trained on the nuisance variables is able to reach even smaller error than on the logits for CE, but just 27.70% error for iCE, indicating that we have successfully removed most of the information of the label from the nuisance variables and fixed the problem of excessive invariance to semantically meaningful variability with no cost in test error.network. The first three stages consist of additive and the last of affine coupling layers. After the final layer we apply an orthogonal 2D DCT type-II to all feature maps and read out the classes in the low-pass components of the transformation. This effectively gives us an invertible global average pooling and makes our network even more similar to ResNets, that always apply global average pooling on their final feature maps. We train the network with momentum SGD for 128 epochs, a batch size of 480 (distributed to 6 GPUs), a base learning rate of 0.1, which is reduced by a factor of 0.1 every 32 epochs. We apply momentum of 0.9 and l2 weight decay of 1e-4.

represent the short summary for finding the corresponding scientific article (long)!

We show deep networks are not only too sensitive to task-irrelevant changes of their input, but also too invariant to a wide range of task-relevant changes, thus making vast regions in input space vulnerable to adversarial attacks.

Represent the short summary for finding the corresponding scientific article (long).

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models. Deep generative models -latent variable models in the form of variational autoencoders BID16, implicit generative models in the form of GANs BID8, and exact likelihood models like PixelRNN/CNN (van den c), Image Transformer BID22, PixelSNAIL, NICE, RealNVP, and Glow BID5 BID15 -have recently begun to successfully model high dimensional raw observations from complex real-world datasets, from natural images and videos, to audio signals and natural language BID14 BID34.Autoregressive models, a certain subclass of exact likelihood models, achieve state-of-the-art density estimation performance on many challenging real-world datasets, but generally suffer from slow sampling time due to their autoregressive structure BID28 BID22. Inverse autoregressive models can sample quickly and potentially have strong modeling capacity, but they cannot be trained efficiently by maximum likelihood. Non-autoregressive flow-based models (which we will refer to as "flow models"), such as NICE, RealNVP, and Glow, are efficient for sampling, but have so far lagged behind autoregressive models in density estimation benchmarks BID5 BID15.In the hope of creating an ideal likelihood-based generative model that simultaneously has fast sampling, fast inference, and strong density estimation performance, we seek to close the density estimation performance gap between flow models and autoregressive models. In subsequent sections, we present our new flow model, Flow++, which is powered by an improved training procedure for continuous likelihood models and a number of architectural extensions of the coupling layer defined by BID5. A flow model f is constructed as an invertible transformation that maps observed data x to a standard Gaussian latent variable z = f (x), as in nonlinear independent component analysis BID1 BID10 BID9. The key idea in the design of a flow model is to form f by stacking individual simple invertible transformations BID5 BID15 BID25 BID19. Explicitly, f is constructed by composing a series of invertible flows as DISPLAYFORM0, with each f i having a tractable inverse and a tractable Jacobian determinant. This way, sampling is efficient, as it can be performed by computing DISPLAYFORM1 1 (z) for z ∼ N (0, I), and so is training by maximum likelihood, since the model density DISPLAYFORM2 is easy to compute and differentiate with respect to the parameters of the flows f i. In this section, we describe three modeling inefficiencies in prior work on flow models: uniform noise is a suboptimal dequantization choice that hurts both training loss and generalization; commonly used affine coupling flows are not expressive enough; convolutional layers in the conditioning networks of coupling layers are not powerful enough. Our proposed model, Flow++, consists of a set of improved design choices: variational flow-based dequantization instead of uniform dequantization; logistic mixture CDF coupling flows; self-attention in the conditioning networks of coupling layers. Many real-world datasets, such as CIFAR10 and ImageNet, are recordings of continuous signals quantized into discrete representations. Fitting a continuous density model to discrete data, however, will produce a degenerate solution that places all probability mass on discrete datapoints BID30. A common solution to this problem is to first convert the discrete data distribution into a continuous distribution via a process called "dequantization," and then model the ing continuous distribution using the continuous density model BID30 BID6 BID28. Dequantization is usually performed in prior work by adding uniform noise to the discrete data over the width of each discrete bin: if each of the D components of the discrete data x takes on values in {0, 1, 2, . . ., 255}, then the dequantized data is given by y = x + u, where u is drawn uniformly from D. BID29 note that training a continuous density model p model on uniformly dequantized data y can be interpreted as maximizing a lower bound on the log-likelihood for a certain discrete model P model on the original discrete data x: DISPLAYFORM0 The argument of BID29 proceeds as follows. Letting P data denote the original distribution of discrete data and p data denote the distribution of uniformly dequantized data, Jensen's inequality implies that DISPLAYFORM1 Consequently, maximizing the log-likelihood of the continuous model on uniformly dequantized data cannot lead to the continuous model degenerately collapsing onto the discrete data, because its objective is bounded above by the log-likelihood of a discrete model. While uniform dequantization successfully prevents the continuous density model p model from collapsing to a degenerate mixture of point masses on discrete data, it asks p model to assign uniform density to unit hypercubes x + D around the data x. It is difficult and unnatural for smooth function approximators, such as neural network density models, to excel at such a task. To sidestep this issue, we now introduce a new dequantization technique based on variational inference. Again, we are interested in modeling D-dimensional discrete data x ∼ P data using a continuous density model p model, and we will do so by maximizing the log-likelihood of its associated discrete model P model (x):= D p model (x + u) du. Now, however, we introduce a dequantization noise distribution q(u|x), with support over u ∈ D. Treating q as an approximate posterior, we have the following variational lower bound, which holds for all q: DISPLAYFORM0 We will choose q itself to be a conditional flow-based generative model of the form u = q x , where DISPLAYFORM1 x /∂u, and thus we obtain the objective DISPLAYFORM2 which we maximize jointly over p model and q. When p model is also a flow model x = f −1 (z) (as it is throughout this paper), it is straightforward to calculate a stochastic gradient of this objective using the pathwise derivative estimator, as f (x + q x ) is differentiable with respect to the parameters of f and q. Notice that the lower bound for uniform dequantization -eqs. to -is a special case of our variational lower bound -eqs. to, when the dequantization distribution q is a uniform distribution that ignores dependence on x. Because the gap between our objective and the true expected log-likelihood DISPLAYFORM3, using a uniform q forces p model to unnaturally place uniform density over each hypercube x + D to compensate for any potential looseness in the variational bound introduced by the inexpressive q. Using an expressive flow-based q, on the other hand, allows p model to place density in each hypercube x + D according to a much more flexible distribution q(u|x). This is a more natural task for p model to perform, improving both training and generalization loss. Recent progress in the design of flow models has involved carefully constructing flows to increase their expressiveness while preserving tractability of the inverse and Jacobian determinant computations. One example is the invertible 1 × 1 convolution flow, whose inverse and Jacobian determinant can be calculated and differentiated with standard automatic differentiation libraries BID15. Another example, which we build upon in our work here, is the affine coupling layer BID6. It is a parameterized flow y = f θ (x) that first splits the components of x into two parts x 1, x 2, and then computes y = (y 1, y 2), given by DISPLAYFORM0 Here, a θ and b θ are outputs of a neural network that acts on x 1 in a complex, expressive manner, but the ing behavior on x 2 always remains an elementwise affine transformation -effectively, a θ and b θ together form a data-parameterized family of invertible affine transformations. This allows the affine coupling layer to express complex dependencies on the data while keeping inversion and log-likelihood computation tractable. Using · and exp to respectively denote elementwise multiplication and exponentiation, DISPLAYFORM1 The splitting operation x → (x 1, x 2) and merging operation (y 1, y 2) → y are usually performed over channels or over space in a checkerboard-like pattern BID6. We found in our experiments that density modeling performance of these coupling layers could be improved by augmenting the data-parameterized elementwise affine transformations by more general nonlinear elementwise transformations. For a given scalar component x of x 2, we apply the cumulative distribution function (CDF) for a mixture of K logistics -parameterized by mixture probabilities, means, and log scales π, µ, s -followed by an inverse sigmoid and an affine transformation parameterized by a and b: DISPLAYFORM0 where MixLogCDF(x; π, µ, DISPLAYFORM1 The transformation parameters π, µ, s, a, b for each component of x 2 are produced by a neural network acting on x 1 . This neural network must produce these transformation parameters for each component of x 2, hence it produces vectors a θ (x 1) and b θ (x 1) and tensors π θ (x 1), µ θ (x 1), s θ (x 1) (with last axis dimension K). The coupling transformation is then given by: DISPLAYFORM2 where the formula for computing y 2 operates elementwise. The inverse sigmoid ensures that the inverse of this coupling transformation always exists: the range of the logistic mixture CDF is, so the domain of its inverse must stay within this interval. The CDF itself can be inverted efficiently with bisection, because it is a monotonically increasing function. Moreover, the Jacobian determinant of this transformation involves calculating the probability density function of the logistic mixtures, which poses no computational difficulty. In addition to improving the expressiveness of the elementwise transformations on x 2, we found it crucial to improve the expressiveness of the conditioning on x 1 -that is, the expressiveness of the neural network responsible for producing the elementwise transformation parameters π, µ, s, a, b. Our best were obtained by stacking convolutions and multi-head self attention into a gated residual network BID20, in a manner resembling the Transformer BID34 with pointwise feedforward layers replaced by 3×3 convolutional layers. Our architecture is defined as a stack of blocks. Each block consists of the following two layers connected in a residual fashion, with layer normalization BID0 after each residual connection: DISPLAYFORM0 where Gate refers to a 1 × 1 convolution that doubles the number of channels, followed by a gated linear unit BID3. The convolutional layer is identical to the one used by PixelCNN++ BID28, and the multi-head self attention mechanism we use is identical to the one in the Transformer BID34. (We always use 4 heads in our experiments, since we found it to be effective early on in our experimentation process.)With these blocks in hand, the network that outputs the elementwise transformation parameters is simply given by stacking blocks on top of each other, and finishing with a final convolution that increases the number of channels to the amount needed to specify the elementwise transformation parameters. Here, we show that Flow++ achieves state-of-the-art density modeling performance among nonautoregressive models on CIFAR10 and 32x32 and 64x64 ImageNet. We also present ablation experiments that quantify the improvements proposed in section 3, and we present example generative samples from Flow++ and compare them against samples from autoregressive models. Our experiments employed weight normalization and data-dependent initialization. We used the checkerboard-splitting, channel-splitting, and downsampling flows of BID6; we also used before every coupling flow an invertible 1x1 convolution flows of BID15, as well as a variant of their "actnorm" flow that normalizes all activations independently (instead of normalizing per channel). Our CIFAR10 model used 4 coupling layers with checkerboard splits at 32x32 resolution, 2 coupling layers with channel splits at 16x16 resolution, and 3 coupling layers with checkerboard splits at 16x16 resolution; each coupling layer used 10 convolution-attention blocks, all with 96 filters. More details on architectures, as well as details for the other experiments, will be given in a source code release. In table 1, we show that Flow++ achieves state-of-the-art density modeling out of all nonautoregressive models, and it is competitive with autoregressive models: its performance is on par with the first generation of PixelCNN models, and it outperforms Multiscale PixelCNN BID24. As of submission, our models have not fully converged due to computational constraint and we expect further performance gain in future revision of this manuscript. 3.14 --PixelRNN (van den 3.00 3.86 3.63 Gated PixelCNN (van den BID33 3.03 3.83 3.57 PixelCNN++ BID28 2.92 --Image Transformer BID22 2.90 3.77 -PixelSNAIL 2.85 3.80 3.52 We ran the following ablations of our model on unconditional CIFAR10 density estimation: variational dequantization vs. uniform dequantization; logistic mixture coupling vs. affine coupling; and stacked self-attention vs. convolutions only. As each ablation involves removing some component of the network, we increased the number of filters in all convolutional layers (and attention layers, if present) in order to match the total number of parameters with the full Flow++ model. In FIG0 and table 2, we compare the performance of these ablations relative to Flow++ at 400 epochs of training, which was not enough for these models to converge, but far enough to see their relative performance differences. Switching from our variational dequantization to the more standard uniform dequantization costs the most: approximately 0.127 bits/dim. The remaining two ablations both cost approximately 0.03 bits/dim: switching from our logistic mixture coupling layers to affine coupling layers, and switching from our hybrid convolution-and-self-attention architecture to a pure convolutional residual architecture. Note that these performance differences are present despite all networks having approximately the same number of parameters: the improved performance of Flow++ comes from improved inductive biases, not simply from increased parameter count. The most interesting is probably the effect of the dequantization scheme on training and generalization loss. At 400 epochs of training, the full Flow++ model with variational dequantization has a train-test gap of approximately 0.02 bits/dim, but with uniform dequantization, the train-test gap is approximately 0.06 bits/dim. This confirms our claim in Section 3.1.2 that training with variational dequantization is a more natural task for the model than training with uniform dequantization. BID23. More samples are available in the appendix (section 7). Likelihood-based models constitute a large family of deep generative models. One subclass of such methods, based on variational inference, allows for efficient approximate inference and sampling, but does not admit exact log likelihood computation BID16 BID26. Another subclass, which we called exact likelihood models in this work, does admit exact log likelihood computation. These exact likelihood models are typically specified as invertible transformations that are parameterized by neural networks BID4 BID18 BID30 BID5 BID7 BID28.There is prior work that aims to improve the sampling speed of deep autoregressive models. The Multiscale PixelCNN BID24 modifies the PixelCNN to be non-fully-expressive by introducing conditional independence assumptions among pixels in a way that permits sampling in a logarithmic number of steps, rather than linear. Such a change in the autoregressive structure allows for faster sampling but also makes some statistical patterns impossible to capture, and hence reduces the capacity of the model for density estimation. WaveRNN BID13 improves sampling speed for autoregressive models for audio via sparsity and other engineering considerations, some of which may apply to flow models as well. There is also recent work that aims to improve the expressiveness of coupling layers in flow models. BID15 demonstrate improved density estimation using an invertible 1x1 convolution flow, and demonstrate that very large flow models can be trained to produce photorealistic faces. BID21 introduce piecewise polynomial couplings that are similar in spirit to our mixture of logistics couplings. They found them to be more expressive than affine couplings, but reported little performance gains in density estimation. We leave a detailed comparison between our coupling layer and the piecewise polynomial CDFs for future work. We presented Flow++, a new flow-based generative model that begins to close the performance gap between flow models and autoregressive models. Our work considers specific instantiations of design principles for flow models -dequantization, flow design, and conditioning architecture design -and we hope these principles will help guide future research in flow models and likelihoodbased models in general.7 APPENDIX A: SAMPLES

Represent the TLDR for finding the corresponding scientific article.

Improved training of current flow-based generative models (Glow and RealNVP) on density estimation benchmarks

Represent the short summary for finding the corresponding scientific article (long)

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.