Question: represent the scientific article to find a short tldr

Question: Adversarial attacks on convolutional neural networks (CNN) have gained significant attention and there have been active research efforts on defense mechanisms. Stochastic input transformation methods have been proposed, where the idea is to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples. However, the transformation improves the accuracy on adversarial images at the expense of the accuracy on clean images. While it is intuitive that the accuracy on clean images would deteriorate, the exact mechanism in which how this occurs is unclear. In this paper, we study the distribution of softmax induced by stochastic transformations. We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the ing distribution of softmax could be used to correct the prediction. Furthermore, on the adversarial counterparts, with the image transformation, the ing shapes of the distribution of softmax are similar to the distributions from the clean images. With these observations, we propose a method to improve existing transformation-based defenses. We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images. Our empirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images. Our method is generic and can be integrated with existing transformation-based defenses. There has been widespread use of convolutional neural networks (CNN) in many critical real-life applications such as facial recognition and self-driving cars . However, it has been found that CNNs could misclassify the input image when the image has been corrupted by an imperceptible change . In other words, CNNs are not robust to small, carefully-crafted image perturbations. Such images are called adversarial examples and there have been active research efforts in designing attacks that show the susceptibility of CNNs. Correspondingly, many defense methods that aim to increase robustness to attacks have been proposed. Stochastic transformation-based defenses have shown considerable success in recovering from adversarial attacks. Under these defenses, the input image is transformed in a certain way before feeding into the CNN, such that the transformed adversarial image would no longer be adversarial. As the transformation is random, by feeding in samples of the transformed image through the CNN, we accumulate a set of CNN softmax outputs and predictions. As such, existing transformationbased defenses take a majority vote of the CNN predictions from the randomly transformed image . Transformation-based defenses are desirable as there is no need to retrain the CNN model. However, they suffer from deterioration of performance on clean images. With increasing number of pixel deflections , there is improvement on the performance on adversarial images, but this comes with a rapid deterioration of performance on clean images. In transformation-based defenses, the image is transformed stochastically where each sample t x is drawn from the distribution T (x) and then fed to the CNN (blue box). In our defense method, for each input image x, we build the marginal distribution of softmax probabilities from the transformed samples t x, · · ·. The distributions are fed to a separate distribution classifier which performs the final classification. Note that our distribution classifier is trained only on distributions obtained from clean images while tested on both clean and adversarial images. The exact mechanism of the deterioration in performance on clean images is unclear. We believe that the softmax distribution induced by the random transformation contains rich information which is not captured by majority vote that simply counts the final class predictions from the transformed samples. Now, an interesting question is whether the features in the distribution of softmax could be better utilized. In this paper, to elucidate how the deterioration in accuracy on clean images occurs, we study the effects of the random image transformations on the distribution of the softmax outputs and make some key observations. After the image transform, some clean images show distributions of softmax with modes at an incorrect class, reflecting the deterioration in voting accuracy as observed before. While the shifting of the distribution mode to the incorrect class is detrimental to the voting prediction, the ing distribution of softmax contains features that is useful for correcting the prediction. In addition, we observe that the adversarial counterparts show similar shifts in the distributions of softmax as the clean images. We also look into the distribution shapes for the transformed clean and adversarial images and find that they are similar. With these observations, we propose a simple method to improve existing transformation-based defenses, as illustrated in Figure 1. We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed clean images and predict the class label. Without retraining the original CNN, our distribution classifier improves the performance of transformation-based defenses on both clean and adversarial images. On the MNIST dataset, the improvements in accuracy over majority voting are 1.7% and 5.9% on the clean and adversarial images respectively. On CIFAR10, the improvements are 6.4% and 3.6% respectively. Note that the distributions obtained from the adversarial images are not included in the training of the distribution classifier. In real-world settings, the type of attack is not known beforehand. Training the distribution classifier on a specific attack may cause the classifier to overfit to that attack. Hence, it is an advantage that our defense method is attack-agnostic. Our experimental findings show that the features of the distribution in the softmax are useful and can be used to improve existing transformation-based defenses. Our contributions are as follows: 1. We analyze the effects of image transformation in existing defenses on the softmax outputs for clean and adversarial images, with a key finding that the distributions of softmax obtained from clean and adversarial images share similar features. 2. We propose a method that trains a distribution classifier on the distributions of the softmax outputs of transformed clean images only, but show improvements in both clean and adversarial images. This method is agnostic to the attack method, does not require retraining of the CNN and can be integrated with existing transformation-based methods. Given an image dataset {(x 1, y 1) · · · (x M, y M)} and a classifier F θ that has been trained with this dataset, with parameters θ, the aim of the attack is to produce an adversarial image x adv i such that F θ (x adv i) = y i, and ||x adv i − x i || is small. We focus on four gradient-based untargeted attacks. The Fast Gradient Sign Method (FGSM) is a single-step attack that uses the sign of the gradient of the classification loss to perturb the image. Iterative Gradient Sign Method (IGM) (a) is an iterative version of FGSM. In DeepFool , at each iteration, the attack approximates the classifier with a linear decision boundary and generates the minimal perturbation to cross the boundary. Finally, the Carlini & Wagner (C&W) L 2 attack jointly minimizes the perturbation L 2 norm and a differentiable loss function based on the classifier's logit outputs. Besides gradient-based attacks, there are also black-box attacks where the CNN model is not known and only the softmax output or final prediction is given (; ;). Defense methods have been proposed to make the classifiers more robust. In adversarial training, the CNN model is trained on adversarial examples generated from itself (; b) or from an ensemble of models (Tramèr et al., 2017). Other methods involve training auxiliary neural networks on mixture of clean and adversarial images, for instance, by denoising the inputs with a neural network before feeding into the CNN (; ;) or by training a neural network on the CNN logits . In the next section, we introduce another class of defense: transformation-based defenses. Transformation-based defenses aim to recover from adversarial perturbations, that is for input transformation T, we want F θ (T (x adv i)) = y i. At the same time, the accuracy on the clean images has to be maintained, ie. F θ (T (x i)) = y i. Note that transformation-based defenses are implemented at test time and this is different from training-time data augmentation. Here we introduce two transformation-based defenses that we experiment on. Pixel deflection (PD) : Pixel deflection corrupts an image by locally redistributing pixels. At each step, it selects a random pixel and replaces it with another randomly selected pixel in a local neighborhood. The probability of a pixel being selected is inversely proportional to the class activation map . Lastly, there is a denoising step based on wavelet transform. In our experiments, we did not use robust activation maps for our datasets as we found that this omission did not cause significant difference in performance (see Appendix D.3). Random resize and padding (RRP) : Each image is first resized to a random size and then padded with zeroes to a fixed size in a random manner. In many transformation-based methods, the transformation is stochastic. Hence there can be different samples of the transformation of an image: t x ∼ T (x), where t x represents a transformed sample. Existing transformation defenses benefit from improved performance by taking the majority vote across samples of random transformations. The advantage of transformation-based methods is that there is no retraining of the CNN classifier. However, a weakness, as identified by , is that the transformation increases the accuracy on adversarial images at the expense of the accuracy on clean images. The exact mechanism of the deterioration in performance on clean images is unclear. In this paper, we elucidate how the deterioration in accuracy on clean images occurs by studying the effects of the random image transformations on the distribution of the softmax outputs. Due to the randomness of the transforms, samples of the transformed image will have different softmax outputs. With each image, we obtain a distribution over the softmax outputs accumulated from multiple samples of the transformation. These are the steps to obtain the distribution of softmax: 1. For each input image x, obtain N transformed samples: The transformed samples of the image (t x) are fed into the CNN individually to obtain their softmax probabilities. Let σ x,j, for j = 1, · · ·, C, be the j-th component of the softmax vector. C denotes the number of classes for the classification task. With each input image and a transformation method, there exists an underlying joint distribution of the CNN softmax probabilities, from which we estimate with N samples. 3. The underlying joint distribution of the softmax has a dimension equal to the number of classes (eg. 10-D for MNIST). Performing accurate density estimation in high dimensions is challenging due to the curse of dimensionality. Here we make an approximation by computing the marginal distributions over each class. When we use the term'distribution of softmax', we are referring to the marginalized distributions. We use kernel density estimation with a Gaussian kernel. Let h x,j be the distribution accumulated from σ where δ is the kernel width and s ∈ is the support of the softmax output. The distribution is then discretized into bins. In this section, we study the effect of image transformation on the distribution of the softmax and make several interesting observations. In the following analyses, we study a LeNet5 CNN trained with MNIST. The adversarial images are generated using FGSM and for the transformation defense, we use pixel deflection, with N =100 transformation samples per image. The image transformation magnitude is controlled by the number of pixel deflections, d. In the analysis here and in the experimental in Section 5, when reporting the accuracies, on clean images, we consider images that have been correctly predicted by the CNN, hence without any transformation defense, the test accuracy is 100%. This is following the setup of where the misclassified images by CNN are excluded as it is not meaningful to evaluate any attack (and subsequent defense) methods on these images. For adversarial images, we consider the images that have been successfully attacked, so the test accuracy reflects the recovery rate and without any defense the accuracy is 0%. In Figure 2a, we show how the image transformation affects the voting predictions on two MNIST classes. For each MNIST class, we take all the clean and adversarial test images, perform the transformation and then feed through the CNN to obtain the final voting prediction. We observe that for class label 8, there is some recovery from the attack as some adversarial images are voted to the correct class after the transformation. However, some clean images get misclassified to other classes (eg. 2 and 3). Although this means there is a deterioration of the accuracy on clean images, it is interesting that the misclassifications have the same voting classes as the transformed adversarial images. A similar pattern is observed for class label 6 where the clean images are misclassified to classes 4 and 5, which overlap with the vote predictions of some adversarial images at d = 300. With the above analysis, we characterize the relationship between the clean and adversarial images in terms of the JS divergence of the distributions of the softmax at increasing number of pixel deflections. For each MNIST digit class, we quantify the distance of the distributions among the clean images (clean-clean, same class), distance of the distributions among the adversarial images (adversarial-adversarial, same class), the distance of the distributions between clean and adversarial images (clean-adversarial, same class) and the distance of the distributions between clean images of this class and all other classes (clean-clean, different class). Here we give details on the calculation of the 4 distance measures. First, the distance between the distributions of softmax output for two input images, x 1 and x 2 is given by, where D JS is the Jensen-Shannon divergence. Distance measures of and computed by taking the average distance of each image distribution to the centroid distribution which is computed with is computed by the distance between the centroids of the clean and adversarial distributions. Finally, is computed by the distance of the centroid distribution of the clean images of the particular class with the centroid distribution of another class, averaged over the other 9 classes. In Figure 2b, we show for two MNIST classes, but similar trends are observed across all classes (see Figure 8 in Appendix A)). The clean-clean (same-class) distance starts off low initially as all clean samples will give high scores at the correct class. With increasing number of deflections, there is increased variability in the softmax outputs and the ing distributions. Next, the adversarial images of the same class are initially predicted as different incorrect classes without any transformation, and hence the adversarial-adversarial (same-class) distance starts off high and decreases with more transformation. The clean-adversarial (same-class) distance decreases with increasing image transformation which shows that the distributions of softmax from the clean and adversarial images are becoming more similar. Finally, the clean-clean (different class) distance decreases as well, which is expected because we already know that with more transformation, the clean image voting accuracy deteriorates. However, we observe that clean-clean (different class) distance decreases less rapidly and remains higher than clean-clean (same-class) distance at d=300. This means the transformation still retains information about the differences between the classes. At d=800, all 4 distance measures converge, which suggests the number of deflections is too large and the differences between the classes are no longer retained. Next, we visualize the morphing of the distributions with increasing number of pixel deflections for an example image in Figure 3. For the purpose of visualization, instead of the per-class marginal distributions of the softmax, we perform kernel density estimation (kde) on the softmax values for the marginals on class 5 and 6. The softmax values of the other 8 classes are not shown. We have not excluded the areas where performing kde in sum probability exceeding one, and our visualization still conveys our ideas and the distribution shapes well. Without any image transformation, as expected, the softmax outputs of the clean and adversarial images are very different. As the number of pixel deflections increases, each point evolves to a distribution due to the randomness of the transformation. The voting mechanism is straightforward; an image is classified to the class where the distribution mass is largest. In this example, the distribution shapes for the clean and adversarial image become more similar, and in the same incorrect voting prediction at d=300. This shows the similarity of distributions obtained from clean and adversarial images after image transformation, which was illustrated in Figure 2b. In Figure 4, we show more examples of the distributions obtained from clean images (A-H) and their adversarial counterparts (Ã-H) at d=300. For clean images A-D, voting predicts correctly but on the adversarial counterpartsÃ-D, voting predicts wrongly. For clean images E-H and the adversarial counterpartsẼ-H, voting predicts wrongly. For completeness, we also show in Figure 9 in Appendix B examples of adversarial images where the transformation defense, coupled with voting, has successfully recovered the correct class. With the random image transformation, there are similarities in the distribution shapes between the clean and adversarial images, as shown by the groupings and arrows (eg. between E andÃ,Ẽ,F). This further supports our earlier observations. After the image transformation, the voting accuracy on the clean images deteriorates, but the ing distributions have similar features as the distributions from the adversarial counterparts. This gives us an idea to enhance existing transformation-based defenses: to train a distribution classifier on the distributions obtained from clean images only, while improving the performance on both clean and adversarial images. Instead of voting, to reduce the drop in performance on clean images, we train a separate compact distribution classifier to recognize patterns in the distributions of softmax probabilities of clean images, as illustrated in Figure 1. For each clean image, the marginal distributions obtained are inputs to the distribution classifier, which learns to associate this distribution with the correct class label. If the individual transformed images were initially misclassified by the CNN, our distribution classifier should learn to recover the correct class. During the test phase, for any input image, clean or adversarial, we build the distribution of softmax from N transformed samples and feed them into our trained distribution classifier to obtain the final prediction. Note that our defense method does not require retraining of the original CNN, is agnostic to the attack method and can be integrated with most existing stochastic transformation-based methods. Distribution classifiers: We investigated three distribution classification methods. First, we adapt a state-of-the-art distribution-to-distribution regression method, called distribution regression network (DRN) . Details of the adaptation of DRN are included in Appendix C. We also experimented on random forest (RF), which alleviates overfitting by averaging the outputs from multiple decision trees. Finally, we experimented on multilayer perceptrons (MLP) which are fully connected neural networks, with a softmax layer for the classification task. For this distribution classification task, we concatenate the distribution bins from the softmax classes into a single input vector for RF and MLP. For DRN and MLP, we use the cross entropy loss and the network architectures and optimization hyperparameters are chosen by cross-validation. For random forest, the Gini impurity is used as the splitting criterion and the number of trees and maximum depth of the trees are tuned by cross-validation. The hyperparameter values are included in Appendix D.4. In the following section, we describe our experimental setup to evaluate the performance on clean and adversarial images with our distribution classifier method. We use the MNIST , CIFAR10 and CIFAR100 et al., 1998 ) that has 98.7% test accuracy. For CIFAR10 and CIFAR100, we use wide ResNet with test accuracies of 95.7% and 78.9% respectively. Attack methods: As introduced in Section 2, we use four adversarial attacks in the untargeted setting. In Appendix D.1, we have included the distortion metrics, the success rates and the hyperparameters. The attacks are implemented using the CleverHans library . Transformation-based defenses: As a baseline, we use a random pixel noise (RPN) as a defense method, where each pixel noise is sampled with a uniform distribution with L ∞ measure. In addition, we use two existing transformation-based methods: pixel deflection (PD) and image random resize and pad (RRP) . Although these two methods have not been tested for MNIST, CIFAR10 and CIFAR100, we find that they work considerably well and present the here. The hyperparameter tuning for each defense is conducted on the validation sets. We select hyperparameters that give the best recovery from adversarial attack, regardless of the deterioration in accuracy on clean images. The hyperparameters are included in Appendix D.2. To test the effectiveness of the transformation-based defenses before integrating with our defense method, we perform majority voting on the transformed image samples. This sets the baseline for our distribution classifier defense method. When reporting the test accuracies, on clean images, we consider images that have been correctly predicted by the CNN, hence without any defense method, the test accuracy is 100%. For adversarial images, we consider the images that have been successfully attacked, so the test accuracy reflects the recovery rate and without any defense the accuracy is 0%. For the MNIST dataset, N = 100 transformation samples were used for voting and for constructing the distribution of softmax. We found that the distribution classifiers required only 1000 training data, which is a small fraction out of the original 50,000 data. Figure 5 (left) shows the test accuracies of the three transformation-based defenses with majority voting and with the three distribution classifiers. Table 11 in Appendix D.5 shows the numerical figures of the in Figure 5. First, we observe that the recovery on adversarial images with majority voting for the iterative methods IGSM, DeepFool and C&W is much better compared to the single-step FGSM. This is in line with the observations made by where they found their defense to be more effective for iterative attacks. The distribution classifiers are trained to reduce the deterioration of accuracy on clean images. The distribution classifiers have improved accuracy over voting on the clean images for most cases, except when the voting accuracy was already high (eg. 100% voting accuracy for PD on DeepFool). The mean improvement of the accuracy on the clean images is 1.7% for DRN. Hence, our distribution classifier method is stronger than voting. Voting simply takes the mode of the softmax probabilities of the transformed image, disregarding properties such as variance across the classes. In contrast, the distribution classifier learns from the distinctive features of the distribution of softmax. Without training on the distributions obtained from adversarial images, our method has managed to improve the recovery rate. The mean improvement of the accuracy on the adversarial images is 5.9% for DRN. The three distribution classifier methods are comparable, except for some cases where DRN outperforms other classifiers (eg. PD adv., IGSM) and where MLP and RF have lower accuracy than voting (eg. RPN adv., DeepFool and C&W). In the earlier section in Figure 4, we show that after image transformation, the distributions of softmax between the clean and adversarial images show some similarities and distinctive features. In fact, all of the clean (A-H) and adversarial (Ã-H) images (class 6) are classified correctly by the distribution classifier. Even though the distribution classifier was only trained on distributions from the clean images (A-H), the distribution classifier can recover the correct class for the adversarial images where voting has failed (Ã-H). The distribution classifier does so by learning the distinctive shapes of the distributions associated with the digit class from the clean images, and is able to apply this to the adversarial images with similar distribution shapes. Furthermore, our distribution classifier is able to pick up subtle differences in the distribution features. Figure 6a shows examples of clean images with class label 5 that are correctly classified by our distribution classifier. It is interesting that although the distribution shapes for adversarial imagesC and G shown in Figure 4 look similar, our distribution classifier is able to distinguish between the shapes for class 5 and 6. We used N =100 transformed samples in our experiments. Hence, the evaluation time will be 100 times longer than the time taken by a single sample. Here we study the effect of the number of samples. Figure 6b and 6c show the classification accuracies for voting and DRN as the number of transformed samples increases. On the clean images, both voting and DRN accuracies improve with more number of samples, with the performance of voting saturating while DRN's performance continues to increase with widening gap. The widening gap shows that a sufficient number of samples is required to capture the features of the distribution of softmax. On the adversarial images, the accuracies stay more of less the same. Although having more transformed samples is beneficial for the performance on clean images, our distribution classifier improves the voting performance regardless of the number of samples. For the CIFAR10 and CIFAR100 datasets, N = 50 image transformation samples and 10,000 training data were used. Figure 5 (middle) shows the for CIFAR10. All three distribution classifiers gave comparable improvements over voting, except for MLP which performs worse than voting for adversarial images with RPN on DeepFool. For CIFAR100 (Figure 5, right), the distribution classifiers mostly show improved performance over voting. There are exceptions where DRN (eg. PD adv., FGSM) and MLP (eg. RPN adv., DeepFool) have lower accuracy than voting. This suggests that for datasets with more classes, random forest may perform better than other classifiers. As explained in Section 3, in the in Figure 5, we have excluded clean images which are misclassified by the CNN and the images where the attack has failed. To check that our method works on these images, we evaluated these images for CIFAR100 with FGSM attack, random resize and padding and random forest classifier. Our in Table 14 in the Appendix show that our distribution classifier method still outperforms majority voting. In this section, we evaluate end-to-end attacks on our distribution classifier method (with DRN) on the MNIST and CIFAR10 datasets. We use the Boundary Attack which is a black-box decision-based attack. We performed the attack on the base CNN classifier (CNN), CNN with pixel deflection and voting (Vote), and CNN with pixel deflection and distribution classifier trained on clean images (DRN). In addition, we trained the distribution classifier on a mixture of distributions obtained from both clean and adversarial images obtained with IGSM on the base CNN, which can be seen as a lightweight adversarial training scheme (DRN LAT) except that the CNN is kept fixed. Finally, we tested the attack on an adversarially-trained CNN (Adv trained CNN) by with allowed perturbations of L ∞ ≤ 0.3. Since the Boundary Attack uses the L 2 measure, the adversarially-trained CNN which uses the L ∞ metric is not expected to perform well for the L 2 metric. For details of our implementation of the Boundary Attack, please refer to Appendix E. Figure 7 shows the mean L 2 of the perturbations over 100 test images, where all models are attacked with maximum 5000 iterations. CNN and the adversarially-trained CNN have very low perturbations. The stochastic models, Vote, DRN and DRN LAT, have much higher perturbations with lower quality adversarial images, and the difficulty of the attack increases in that order. This shows that the distribution classifier and the lightweight adversarial training extension are more difficult to attack by the Boundary Attack method compared to voting. have shown that under the white-box setting where the attacker has full knowledge of the CNN model and the defense, random transformation defenses are susceptible to further attack by estimating the gradients using multiple transformation samples, in a method called Expectation over Transformation(EOT). To employ white-box attack on our distribution classifier method, there are a few potential challenges. First, our method uses 50 to 100 transformation samples per image to accumulate the distribution of softmax before feeding into the distribution classifier. Attacking our method with EOT will be very time-consuming as it requires taking multiple batches of transformations, each with 50-100 samples. Next, we have shown our method works with different distribution classifier models, including the non-differentiable random forest classifier. While there have been attacks proposed for random forests , it is unclear how feasible it is to combine these attacks with EOT. We leave the evaluation of white-box attacks on our distribution classifier method for future work. Adversarial attacks on convolutional neural networks have gained significant research attention and stochastic input transformation defenses have been proposed. However, with transformation-based defenses, the performance on clean images deteriorates and the exact mechanism in which how this happens is unclear. In this paper, we conduct in-depth analysis on the effects of stochastic transformation-based defenses on the softmax outputs of clean and adversarial images. We observe that after image transformation, the distributions of softmax obtained from clean and adversarial images share similar distinct features. Exploiting this property, we propose a method that trains a distribution classifier on the distributions of the softmax outputs of transformed clean images only, but show improvements in both clean and adversarial images over majority voting. In our current work, we have considered untargeted attacks on the CNN and it is interesting to test our distribution classifier method with targeted attacks. In Section 3, we studied the 4 distance metrics for the distribution of softmax. Figure 8 shows the distance metrics for all ten MNIST classes with increasing number of pixel deflections. In Figure 9, we show examples where pixel deflection with voting recovers from the adversarial attack. For one of the distribution classifier methods, we adapt a state-of-the-art distribution-to-distribution regression method, called distribution regression network (DRN) . DRN encodes an entire distribution in each network node and this compact representation allows it to achieve higher prediction accuracies for the distribution regression task compared to conventional neural networks. Since DRN shows superior regression performance, we adapt DRN for distribution classification in this work. Our adaption of the distribution classifier is shown on the right of Figure 10. The network consists of fully-connected layers, where each node encodes a distribution. The number of hidden layers and nodes per hidden layer are chosen by cross validation. The number of discretization bins for each distribution for the input layer and hidden layers is also tuned as hyperparameters. To adapt DRN for our distribution classification task, for the final layer, we have C nodes representing each class and we use 2 bins for each distribution to represent the logit output for the corresponding class. The cost function for the distribution classifier is the cross entropy loss on the logits. The distribution classifier is optimized by backpropagation using the Adam optimizer . The weight initialization method follows , where the weights are sampled from a uniform random distribution. Tables 1 to 3 show the hyperparameter settings used for the adversarial attacks. The attacks are implemented using the CleverHans library . For DeepFool and C&W, the other hyperparameters used are the default values set in CleverHans. For L 2 norm, we use the root-mean-square distortion normalized by total number of pixels, following previous works. Tables 4 to 6 show the image transformation parameters used for MNIST and CIFAR10 respectively. The hyperparameter tuning for each defense method is conducted on the validation set for each dataset. We select hyperparameters that give the best recovery from adversarial attack, regardless of the deterioration in accuracy on clean images. The pixel deflection defense uses class activation maps (CAMs) to randomly select pixels to undergo the deflection step. In our experiments, we did not use class activation maps and instead randomly select pixels with equal probabilities. First, for the MNIST dataset, CAMs are unsuitable because the LeNet architecture does not have global average pooling layers which are required for CAMs. For the CIFAR10 dataset, the wide ResNet architecture uses a final layer of global average pooling and so we tested CAMs on it. Table 7 compares the performance on clean and adversarial images using the FGSM and IGSM attacks, with and without CAMs, which shows that using CAMs does Table 4: Details of the image transformation parameters for MNIST. The three transformation-based methods tested are random pixel noise (RPN), pixel deflection (PD) and random resize and padding (RRP). For RPN, the noise magnitude is unnormalized (out of 255). For PD, d is the number of deflections, w is the window size and σ is the denoising parameter., w=20, σ=0 d=100, w=20, σ=0 d=10, w=20, σ=0.08 d=100, w=25, σ=0.08 RRP resize range= resize range= resize range= resize range= not cause significant difference in performance. This may be because CAMs are more effective on larger images such as those in ImageNet where there are many more pixels. Our defense method uses a distribution classifier to train on distributions of softmax probabilities obtained from transformed samples of the clean images. For each image, we build the marginal distributions of the softmax for each class using kernel density estimation with a Gaussian kernel. The kernel width is optimized to be 0.05. For DRN and MLP, the network architecture of the distribution classifier and optimization hyperparameters are chosen by cross-validation. For random forest, the number of trees and maximum depth of the trees are tuned by cross-validation. The hyperparameters used are shown in Tables 8 to 10. Here we include the detailed numerical figures for the accuracies of majority voting and the distribution classifier methods. Tables 11 to 13 show the clean and adversarial test accuracies and the 4 attack methods and the 3 defense methods. For Vote, DRN and DRN LAT, the model outputs are random because of the random image transformation. At each step of Boundary Attack, we allow the attack to query the model once, and this involves taking 50-100 transformed samples for the image to perform voting or to feed to the MNIST RPN 1x10 2x10 1x10 1x10 PD 1x10 1x10 1x20 1x20 RRP 1x10 1x10 1x20 1x20 CIFAR10 RPN 1x10 1x20 1x20 1x20 PD 1x20 1x10 1x20 1x10 RRP 1x20 1x10 1x20 1x10 CIFAR100 RPN 1x100 1x100 1x100 1x100 PD 1x100 1x100 1x100 1x100 RRP 1x100 1x100 1x100 1x100 MNIST RPN 1x20 1x20 1x50 1x20 PD 1x20 1x50 1x50 1x50 RRP 1x50 1x20 1x50 1x50 CIFAR10 RPN 1x50 1x50 1x50 1x20 PD 1x50 1x50 1x50 1x50 RRP 1x50 1x50 1x20 1x20 CIFAR100 RPN 1x2000 1x1000 1x500 1x500 PD 1x500 1x1000 2x100 1x1000 RRP 1x200 1x1000 2x100 1x1000 MNIST RPN n=100, d=20 n=100, d=20 n=200, d=50 n=200, d=50 PD n=100, d=20 n=100, d=20 n=200, d=50 n=200, d=50 RRP n=200, d=20 n=100, d=20 n=200, d=50 n=200, d=50 CIFAR10 RPN n=200, d=50 n=200, d=50 n=200, d=50 n=200, d=50 PD n=200, d=50 n=200, d=50 n=200, d=50 n=200, d=50 RRP n=200, d=50 n=200, d=50 n=200, d=50 n=200, d=50 CIFAR100 RPN n=200, d=200 n=200, d=200 n=200, d=200 n=200, d=200 PD n=200, d=200 n=200, d=200 n=200, d=100 n=200, d=200 RRP n=200, d=200 n=200, d=200 n=200, d=100 n=200, d=100 Table 13: CIFAR100 : For each attack, we compare the clean and adversarial (adv.) test accuracies with majority voting (Vote) and the three distribution classifier methods: distribution regression network (DRN), random forest (RF) and multilayer perceptron (MLP). The three transformation-based defenses are random pixel noise (RPN), pixel deflection (PD) and random resize and padding (RRP). With no defense, the clean accuracy is 100% and the adversarial accuracy is 0%. distribution classifier to obtain a prediction. To avoid overfitting to a fixed transformation pattern, the transformation is random at each step. Our criteria for an image being adversarial is that out of 5 queries, the image is misclassified at least once. Because of the randomness of the model, the image returned by Boundary Attack may be classified to the correct class, and we increase the perturbation by increasing amounts until the image is misclassified. Note that to overcome the randomness, we could have performed multiple queries at each attack step, but because our models already use 50-100 transformed samples per query, this will be computationally infeasible.

We enhance existing transformation-based defenses by using a distribution classifier on the distribution of softmax obtained from transformed images.

represent this text to find a much much longer article about it with say hundreds-thousands of words

E.g.:
We propose new tensor decompositions and associated regularizers to obtain state of the art performances on temporal knowledge base completion. == Most algorithms for representation learning and link prediction in relational data have been designed for static data. However, the data they are applied to usually evolves with time, such as friend graphs in social networks or user interactions with items in recommender systems. This is also the case for knowledge bases, which contain facts such as (US, has president, B. Obama,) that are valid only at certain points in time. For the problem of link prediction under temporal constraints, i.e., answering queries of the form (US, has president, ?, 2012), we propose a solution inspired by the canonical decomposition of tensors of order 4. We introduce new regularization schemes and present an extension of ComplEx that achieves state-of-the-art performance. Additionally, we propose a new dataset for knowledge base completion constructed from Wikidata, larger than previous benchmarks by an order of magnitude, as a new reference for evaluating temporal and non-temporal link prediction methods. Link prediction in relational data has been the subject of interest, given the widespread availability of such data and the breadth of its use in bioinformatics , recommender systems or Knowledge Base completion (a). Relational data is often temporal, for example, the action of buying an item or watching a movie is associated to a timestamp. Some medicines might not have the same adverse side effects depending on the subject's age. The task of temporal link prediction is to find missing links in graphs at precise points in time. In this work, we study temporal link prediction through the lens of temporal knowledge base completion, which provides varied benchmarks both in terms of the underlying data they represent, but also in terms of scale. A knowledge base is a set of facts (subject, predicate, object) about the world that are known to be true. Link prediction in a knowledge base amounts to answer incomplete queries of the form (subject, predicate, ?) by providing an accurate ranking of potential objects. In temporal knowledge bases, these facts have some temporal metadata attached. For example, facts might only hold for a certain time interval, in which case they will be annotated as such. Other facts might be event that happened at a certain point in time. Temporal link prediction amounts to answering queries of the form (subject, predicate, ?, timestamp). For example, we expect the ranking of queries (USA, president, ?, timestamp) to vary with the timestamps. As tensor factorization methods have proved successful for Knowledge Base Completion (a; ;), we express our Temporal Knowledge Base Completion problem as an order 4 tensor completion problem. That is, timestamps are discretized and used to index a 4-th mode in the binary tensor holding (subject, predicate, object, timestamps) facts. First, we introduce a ComplEx decomposition of this order 4 tensor, and link it with previous work on temporal Knowledge Base completion. This decomposition yields embeddings for each timestamps. A natural prior is for these timestamps representation to evolve slowly over time. We are able to introduce this prior as a regularizer for which the optimum is a variation on the nuclear p-norm. In order to deal with heterogeneous temporal knowledge bases where a significant amount of relations might be non-temporal, we add a non-temporal component to our decomposition. Experiments on available benchmarks show that our method outperforms the state of the art for similar number of parameters. We run additional experiments for larger, regularized models and obtain improvements of up to 0.07 absolute Mean Reciprocal Rank (MRR). Finally, we propose a dataset of 400k entities, based on Wikidata, with 7M train triples, of which 10% contain temporal validity information. This dataset is larger than usual benchmarks in the Knowledge Base completion community and could help bridge the gap between the method designed and the envisaged web-scale applications. Matrices and tensors are upper case letters. The i-th row of U is denoted by u i while it's j − th column is denoted by U:,j. The tensor product of two vectors is written ⊗ and the hadamard (elementwise) product ⊙. Static link prediction methods Standard tensor decomposition methods have lead to good (; ; ; Balažević et al., 2019) in Knowledge Base completion. The Canonical Polyadic (CP) Decomposition is the tensor equivalent to the low-rank decomposition of a matrix. A tensor X of canonical rank R can be written as: Setting U = W leads to the Distmult model, which has been successful, despite only being able to represent symmetric score functions. In order to keep the parameter sharing scheme but go beyond symmetric relations, use complex parameters and set W to the complex conjugate of U, U. Regularizing this algorithm with the variational form of the tensor nuclear norm as well as a slight transformation to the learning objective (also proposed in) leads to state of the art in. Other methods are not directly inspired from classical tensor decompositions. For example, TransE models the score as a distance of the translated subject to an object representation. This method has lead to many variations (; ;), but is limited in the relation systems it can model and does not lead to state of the art performances on current benchmarks. propose to generate the entity embeddings of a CP-like tensor decomposition by running a forward pass of a Graph Neural Network over the training Knowledge Base. The experiments included in this work did not lead to better link prediction performances than the same decomposition (Distmult) directly optimized . describes a bayesian model and learning method for representing temporal relations. The temporal smoothness prior used in this work is similar to the gradient penalty we describe in Section 3.3. However, learning one embedding matrix per timestamp is not applicable to the scales considered in this work. uses a tensor decomposition called ASALSAN to express temporal relations. This decomposition is related to RESCAL which underperforms on recent benchmarks due to overfitting (b). For temporal knowledge base completion, learns entity embeddings that change over time, by masking a fraction of the embedding weights with an activation function of learned frequencies. Based on the Tucker decomposition, ConT learns one new core tensor for each timestamp. Finally, viewing the time dimension as a sequence to be predicted, García-Durán et al. use recurrent neural nets to transform the embeddings of standard models such as TransE or Distmult to accomodate the temporal data. Published as a conference paper at ICLR 2020 by studying and extending a regularized CP decomposition of the training set seen as an order 4 tensor. We propose and study several regularizer suited to our decompositions. In this section, we are given facts (subject, predicate, object) annotated with timestamps, we discretize the timestamp range (eg. by reducing timestamps to years) in order to obtain a training set of 4-tuple (subject, predicate, object, time) indexing an order 4 tensor. We will show in Section 5.1 how we reduce each datasets to this setting. , we minimize, for each of the train tuples (i, j, k, l), the instantaneous multiclass loss: Note that this loss is only suited to queries of the type (subject, predicate, ?, time), which is the queries that were considered in related work. We consider another auxiliary loss in Section 6 which we will use on our Wikidata dataset. For a training set S (augmented with reciprocal relations ), and parametric tensor estimateX(θ), we minimize the following objective, with a weighted regularizer Ω: ]. The ComplEx decomposition can naturally be extended to this setting by adding a new factor T, we then have: We call this decomposition TComplEx. Intuitively, we added timestamps embedding that modulate the multi-linear dot product. Notice that the timestamp can be used to equivalently modulate the objects, predicates or subjects to obtain time-dependent representation: Contrary to DE-SimplE , we do not learn temporal embeddings that scale with the number of entities (as frequencies and biases), but rather embeddings that scale with the number of timestamps. The number of parameters for the two models are compared in Table 1. Some predicates might not be affected by timestamps. For example, Malia and Sasha will always be the daughters of Barack and Michelle Obama, whereas the "has occupation" predicate between two entities might very well change over time. In heterogeneous knowledge bases, where some predicates might be temporal and some might not be, we propose to decompose the tensorX as the sum of two tensors, one temporal, and the other non-temporal: Published as a conference paper at ICLR 2020 We call this decomposition TNTComplEx. suggests another way of introducing a non-temporal component, by only allowing a fraction γ of components of the embeddings to be modulated in time. By allowing this sharing of parameters between the temporal and non-temporal part of the tensor, our model removes one hyperparameter. Moreover, preliminary experiments showed that this model outperforms one without parameter sharing. Any order 4 tensor can be considered as an order 3 tensor by unfolding modes together. For a tensor X ∈ R N1×N2×N3×N4, unfolding modes 3 and 4 together will lead to tensorX ∈ R N1×N2×N3N4 . We can see both decompositions ( and) as order 3 tensors by unfolding the temporal and predicate modes together. Considering the decomposition implied by these unfoldings (see Appendix 8.1) leads us to the following weighted regularizers : The first regularizer weights objects, predicates and pairs (predicate, timestamp) according to their respective marginal probabilities. This regularizer is a variational form of the weighted nuclear 3-norm on an order 4 tensor (see subsection 3.4 and Appendix 8.3 for details and proof). The second regularizer is the sum of the nuclear 3 penalties on tensors We have more a priori structure on the temporal mode than on others. Notably, we expect smoothness of the application i → t i. In words, we expect neighboring timestamps to have close representations. Thus, we penalize the norm of the discrete derivative of the temporal embeddings: We show in Appendix 8.2 that the sum of Λ p and the variational form of the nuclear p norm lead to a variational form of a new tensor atomic norm. As was done in , we aim to use tensor nuclear p-norms as regularizers. The definition of the nuclear p-norm of a tensor of order D is: This formulation of the nuclear p-norm writes a tensor as a sum over atoms which are the rank-1 tensors of unit p-norm factors. The nuclear p-norm is NP-hard to compute . , a practical solution is to use the equivalent formulation of nuclear p-norm using their variational form, which can be conveniently written for p = D: For the equality above to hold, the infimum should be over all possible R. The practical solution is to fix R to the desired rank of the decomposition. Using this variational formulation as a regularizer leads to state of the art for order-3 tensors and is convenient in a stochastic gradient setting because it separates over each model coefficient. In addition, this formulation makes it easy to introduce a weighting as recommended in;. In order to learn under non-uniform sampling distributions, one should penalize the weighted norm: In subsection 3.3, we add another penalty in Equation which changes the norm of our atoms. In subsection 3.2, we introduced another variational form in Equation which allows to easily penalize the nuclear 3-norm of an order 4 tensor. This regularizer leads to different weighting. By considering the unfolding of the timestamp and predicate modes, we are able to weight according to the joint marginal of timestamps and predicates, rather than by the product of the marginals. This can be an important distinction if the two are not independent. We study the impact of regularization on the ICEWS05-15 dataset, for the TNTComplEx model. For details on the experimental set-up, see Section 5.1. The first effect we want to quantify is the effect of the regularizer Λ p. We run a grid search for the strength of both Λ p and Ω 3 and plot the convex hull as a function of the temporal regularization strength. As shown in Figure 1, imposing smoothness along the time mode brings an improvement of over 2 MRR point. The second effect we wish to quantify is the effect of the choice of regularizer Ω. A natural regularizer for TNTComplEx would be: We compare ∆ 4, ∆ 3 and ∆ 2 with Ω 3. The comparison is done with a temporal regularizer of 0 to reduce the experimental space. 2 is the common weight-decay frequently used in deep-learning. Such regularizers have been used in knowledge base completion (; 2016b;), however, showed that the infimum of this penalty is non-convex over tensors. 3 matches the order used in the Ω 3 regularizer, and in previous work on knowledge base completion . However, by the same arguments, its minimization does not lead to a convex penalty over tensors. There are two differences between ∆ 4 and Ω 3. First, whereas the first is a variational form of the nuclear 4-norm, the second is a variational form of the nuclear 3-norm which is closer to the nuclear 2-norm. Results for exact recovery of tensors have been generalized to the nuclear 2-norm, and to the extent of our knowledge, there has been no formal study of generalization properties or exact recovery under the nuclear p-norm for p greater than two. Second, the weighting in ∆ 4 is done separately over timestamps and predicates, whereas it is done jointly for Ω 3. This leads to using the joint empirical marginal as a weighting over timestamps and predicates. The impact of weighting on the guarantees that can be obtained are described more precisely in. The contribution of all these regularizers over a non-regularized model are summarized in Table 3. Note that careful regularization leads to a 0.05 MRR increase. A dataset based on Wikidata was proposed by García-Durán et al.. However, upon inspection, this dataset contains numerical data as entities, such as ELO rankings of chess players, which are not representative of practically useful link prediction problems. Also, in this dataset, temporal informations is specified in the form of "OccursSince" and "OccursUntil" statements appended to triples, which becomes unwieldy when a predicate holds for several intervals in time. Moreover, this dataset contains only 11k entities and 150k which is insufficient to benchmark methods at scale.. In order to adress these limitations, we created our own dataset from Wikidata, which we make available at dataseturl. Starting from Wikidata, we removed all entities that were instance of scholarly articles, proteins and others. We also removed disambiguation, template, category and project pages from wikipedia. Then, we removed all facts for which the object was not an entity. We iteratively filtered out entities that had degree at least 5 and predicates that had at least 50 occurrences. With this method, we obtained a dataset of 432715 entities, 407 predicates and 1724 timestamps (we only kept the years). Each datum is a triple (subject, predicate, object) together a timestamp range (begin, end) where begin, end or both can be unspecified. Our train set contains 7M such tuples, with about 10% partially specified temporal tuples. We kept a validation and test set of size 50k each. At train and test time, for a given datum (subject, predicate, object, [begin, end] ), we sample a timestamp (appearing in the dataset) uniformly at random, in the range [begin, end]. For datum without a temporal range, we sample over the maximum date range. Then, we rank the objects for the partial query (subject, predicate, ?, timestamp). We follow the experimental set-up in García-Durán et al.;. We use models from García-Durán et al. and as baselines since they are the best performing algorithms on the datasets considered. We report the filtered Mean Reciprocal Rank (MRR) defined in Nickel et al. (2016b). In order to obtaiqn comparable , we use Table 1 and dataset statistics to compute the rank for each (model, dataset) pair that matches the number of parameters used in. We also report at ranks 10 times higher. This higher rank set-up gives an estimation of the best possible performance attainable on these datasets, even though the dimension used might be impractical for applied systems. All our models are optimized with Adagrad , with a learning rate of 0.1, a batch-size of 1000. More details on the grid-search, actual ranks used and hyper-parameters are given in Appendix 8. (García-Durán et al., 2018) and DE-SimplE are the best numbers reported in the respective papers. Our models have as many parameters as DE-SimplE. Numbers in parentheses are for ranks multiplied by 10. We give on 3 datasets previously used in the litterature: ICEWS14, ICEWS15-05 and Yago15k. The ICEWS datasets are samplings from the Integrated Conflict Early Warning System (ICEWS) 1.García-Durán et al. introduced two subsampling of this data, ICEWS14 which contains all events occuring in 2014 and ICEWS05-15 which contains events occuring between 2005 and 2015. These datasets immediately fit in our framework, since the timestamps are already discretized. The Yago15K dataset (García-Durán et al., 2018) is a modification of FB15k which adds "occursSince" and "occursUntil" timestamps to each triples. Following the evaluation setting of García-Durán et al., during evaluation, the incomplete triples to complete are of the form (subject, predicate, ?, occursSince | occursUntil, timestamp) (with reciprocal predicates). Rather than deal with tensors of order 5, we choose to unfold the (occursSince, occursUntil) and the predicate mode together, multiplying its size by two. Some relations in Wikidata are highly unbalanced (eg. (?, InstanceOf, Human)). For such relations, a ranking evaluation would not make much sense. Instead, we only compute the Mean Reciprocal Rank for missing right hand sides, since the data is such that highly unbalanced relations happen on the left-hand side. However, we follow the same training scheme as for all the other dataset, including reciprocal relations in the training set. The cross-entropy loss evaluated on 400k entities puts a restriction on the dimensionality of embeddings at about d = 100 for a batch-size of 1000. We leave sampling of this loss, which would allow for higher dimensions to future work. We compare ComplEx with the temporal versions described in this paper. We report in Table 2. Note that ComplEx has performances that are stable through a tenfold increase of its number of parameters, a rank of 100 is enough to capture the static information of these datasets. For temporal models however, the performance increases a lot with the number of parameters. It is always beneficial to allow a separate modeling of non-temporal predicates, as the performances of TNTComplex show. Finally, our model match or beat the state of the art on all datasets, even at identical number of parameters. Since these datasets are small, we also report for higher ranks (10 times the number of parameters used for DE-SimplE). outweighs the Temporal MRR (T-MRR). A breakdown of the performances is available in table 4. TNTComplEx obtains performances that are comparable to ComplEx on non-temporal triples, but are better on temporal triples. Moreover, TNTComplEx can minimize the temporal cross-entropy and is thus more flexible on the queries it can answer. Training TNTComplEx on Wikidata with a rank of d = 100 with the full cross-entropy on a Quadro GP 100, we obtain a speed of 5.6k triples per second, leading to experiments time of 7.2 hours. This is to be compared with 5.8k triples per second when training ComplEx for experiments time of 6.9 hours. The additional complexity of our model does not lead to any real impact on runtime, which is dominated by the computation of the cross-entropy over 400k entities. The instantaneous loss described in equation, along with the timestamp sampling scheme described in the previous section only enforces correct rankings along the "object" tubes of our order-4 tensor. In order to enforce a stronger temporal consistency, and be able to answer queries of the type (subject, predicate, object, ?), we propose another cross-entropy loss along the temporal tubes: We optimize the sum of ℓ defined in Equation 1 andl defined in Equation 7. Doing so, we only lose 1 MRR point overall. However, we make our model better at answering queries along the time axis. The macro area under the precision recall curve is 0.92 for a TNTComplEx model learned with ℓ alone and 0.98 for a TNTComplEx model trained with ℓ +l. We plot in Figure 2 the scores along time for train triples (president of the french republic, office holder, {Jacques Chirac | Nicolas Sarkozy | François Hollande | Emmanuel Macron}, ). The periods where a score is highest matches closely the ground truth of start and end dates of these presidents mandates which is represented as a colored . This shows that our models are able to learn rankings that are correct along time intervals despite our training method only ever sampling timestamps within these intervals. Tensor methods have been successful for Knowledge Base completion. In this work, we suggest an extension of these methods to Temporal Knowledge Bases. Our methodology adapts well to the various form of these datasets: point-in-time, beginning and endings or intervals. We show that our methods reach higher performances than the state of the art for similar number of parameters. For several datasets, we also provide performances for higher dimensions. We hope that the gap between low-dimensional and high-dimensional models can motivate further research in models that have increased expressivity at lower number of parameters per entity. Finally, we propose a large scale temporal dataset which we believe represents the challenges of large scale temporal completion in knowledge bases. We give performances of our methods for low-ranks on this dataset. We believe that, given its scale, this dataset could also be an interesting addition to non-temporal knowledge base completion. Then according to , unfolding along modes 3 and 4 leads to an order three tensor of decompositionX Where • is the Khatri-Rao product , which is the column-wise Kronecker product: Note that for a fourth mode of size L: This justifies the regularizers used in Section 3.2. Consider the penalty: ∥ · ∥ τ 4 is a norm and lets us rewrite: ). Following the proof in which only uses homogeneity of the norms, we can show that Ω(U, V, W, T) is a variational form of an atomic norm with atoms: We consider the regularizer: ). Let D subj (resp. obj, pred/time) the diagonal matrix containing the cubic-roots of the marginal probabilities of each subject (resp. obj, pred/time) in the dataset. We denote by • the Kathri-Rao product between two matrices (the columnwise Kronecker product). Summing over the entire dataset, we obtain the penalty: ). Dropping the weightings to simplify notations, we state the equivalence between this regularizer and a variational form of the nuclear 3-norm of an order 4 tensor: The proof follows , noting that ∥u Statistics of all the datasets used in this work are gathered in Table 6: Results for TA (García-Durán et al., 2018) and DE-SimplE are the best numbers reported in the respective papers.

Batch normalization reduces adversarial robustness, as well as general robustness in many cases, particularly to noise corruptions.