Question: represent the text to find a very short summary (1-2 sentences max)

Question: In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize. The human vision system is robust in ways that existing computer vision systems are not BID50 BID1. Unlike current deep learning classifiers BID36 BID21 BID60, the human vision system is not fooled by small changes in query images. Humans are also not confused by many forms of corruption such as snow, blur, pixelation, and novel combinations of these. Humans can even deal with abstract changes in structure and style. Achieving these kinds of robustness is an important goal for computer vision and machine learning. It is also essential for creating deep learning systems that can be deployed in safety-critical applications. Most work on robustness in deep learning methods for vision has focused on the important challenges of robustness to adversarial examples BID54 BID5, unknown unknowns BID25 BID23 BID41, and model or data poisoning BID53. In contrast, we develop and validate datasets for two other forms of robustness. Specifically, we introduce the IMAGETNET-C dataset for input corruption robustness and the IMAGENET-P dataset for input perturbation robustness. To create IMAGENET-C, we introduce a set of 75 common visual corruptions and apply them to the ImageNet object recognition challenge BID7. We hope that this will serve as a general dataset for benchmarking robustness to image corruptions and prevent methodological problems such as moving goal posts and cherry picking. We evaluate the performance of current deep learning systems and show that there is wide room for improvement on IMAGENET-C. We also introduce a total of three methods and architectures that improve corruption robustness without losing accuracy. To create IMAGENET-P, we introduce a set of perturbed or subtly differing ImageNet images. Using metrics we propose, we measure the stability of the network's predictions on these perturbed images. Although these perturbations are not chosen by an adversary, currently existing networks exhibit surprising instability on common perturbations. Then we then demonstrate that approaches which enhance corruption robustness can also improve perturbation robustness. For example, some recent architectures can greatly improve both types of robustness. More, we show that the Adversarial Logit Pairing ∞ adversarial example defense can yield substantial robustness gains on diverse and common perturbations. By defining and benchmarking perturbation and corruption robustness, we facilitate research that can be overcome by future networks which do not rely on spurious correlations or cues inessential to the object's class. Adversarial Examples. An adversarial image is a clean image perturbed by a small distortion carefully crafted to confuse a classifier. These deceptive distortions can occasionally fool black-box classifiers BID38. Algorithms have been developed that search for the smallest additive distortions in RGB space that are sufficient to confuse a classifier. Thus adversarial distortions serve as type of worst-case analysis for network robustness. Its popularity has often led "adversarial robustness" to become interchangeable with "robustness" in the literature BID2. In the literature, new defenses BID42 BID47 BID44 BID22 often quickly succumb to new attacks BID12 BID5, with some exceptions for perturbations on small images BID52. For some simple datasets, the existence of any classification error ensures the existence of adversarial perturbations of size O(d −1/2), d the input dimensionality BID18. For some simple models, adversarial robustness requires an increase in the training set size that is polynomial in d. BID17 suggest modifying the problem of adversarial robustness itself for increased real-world applicability. Robustness in Speech. Speech recognition research emphasizes robustness to common corruptions rather than worst-case, adversarial corruptions BID39 BID45. Common acoustic corruptions (e.g., street noise, chatter, wind) receive greater focus than adversarial audio, because common corruptions are ever-present and unsolved. There are several popular datasets containing noisy test audio BID27 BID26. Robustness in noisy environments requires robust architectures, and some research finds convolutional networks more robust than fully connected networks BID0. Additional robustness has been achieved through pre-processing techniques such as standardizing the statistics of the input BID40 BID58 BID20 BID35. Studies. Several studies demonstrate the fragility of convolutional networks on simple corruptions. For example, BID28 apply impulse noise to break Google's Cloud Vision API. Using Gaussian noise and blur, BID9 demonstrate the superior robustness of human vision to convolutional networks, even after networks are fine-tuned on Gaussian noise or blur. BID15 compare networks to humans on noisy and elastically deformed images. They find that fine-tuning on specific corruptions does not generalize and that classification error patterns underlying network and human predictions are not similar. BID56; propose different corrupted datasets for object and traffic sign recognition. Robustness Enhancements. In an effort to reduce classifier fragility, BID59 finetune on blurred images. They find it is not enough to fine-tune on one type of blur to generalize to other blurs. Furthermore, fine-tuning on several blurs can marginally decrease performance. BID61 also find that fine-tuning on noisy images can cause underfitting, so they encourage the noisy image softmax distribution to match the clean image softmax. BID8 address underfitting via a mixture of corruption-specific experts assuming corruptions are known beforehand. We now define corruption and perturbation robustness and distinguish them from adversarial perturbation robustness. To begin, we consider a classifier f: X → Y trained on samples from distribution D, a set of corruption functions C, and a set of perturbation functions E. We let P C (c), P E (ε) approximate the real-world frequency of these corruptions and perturbations. Most classifiers are judged by their accuracy on test queries drawn from D, i.e., P (x,y)∼D (f (x) = y). Yet in a vast range of cases the classifier is tasked with classifying low-quality or corrupted inputs. In view of this, we suggest also computing the classifier's corruption robustness E c∼C [P (x,y)∼D (f (c(x) = y))]. This contrasts with a popular notion of adversarial robustness, often formulated min δ p <b P (x,y)∼D (f (x + δ) = y), b a small budget. Thus, corruption robustness measures the classifier's average-case performance on corruptions C, while adversarial robustness measures the worst-case performance on small, additive, classifier-tailored perturbations. Average-case performance on small, general, classifier-agnostic perturbations motivates us to define perturbation robustness, namely E ε∼E [P (x,y)∼D (f (ε(x)) = f (x))]. Consequently, in measuring perturbation robustness, we track the classifier's prediction stability, reliability, or consistency in the face of minor input changes. Now in order to approximate C, E and these robustness measures, we designed a set of corruptions and perturbations which are frequently encountered in natural images. We will refer to these as "common" corruptions and perturbations. These common corruptions and perturbations are available in the form of IMAGENET-C and IMAGENET-P.4 THE IMAGENET-C AND IMAGENET-P ROBUSTNESS BENCHMARKS 4.1 THE DATA OF IMAGENET-C AND IMAGENET-P IMAGENET-C Design. The IMAGENET-C benchmark consists of 15 diverse corruption types applied to validation images of ImageNet. The corruptions are drawn from four main categoriesnoise, blur, weather, and digital-as shown in Figure 1. Research that improves performance on this benchmark should indicate general robustness gains, as the corruptions are diverse and numerous. Each corruption type has five levels of severity since corruptions can manifest themselves at varying intensities. Appendix A gives an example of the five different severity levels for impulse noise. Real-world corruptions also have variation even at a fixed intensity. To simulate these, we introduce variation for each corruption when possible. For example, each fog cloud is unique to each image. These algorithmically generated corruptions are applied to the ImageNet BID7 ) validation images to produce our corruption robustness dataset IMAGENET-C. The dataset can be downloaded or re-created by visiting https://github.com/hendrycks/robustness. IMAGENET-C images are saved as lightly compressed JPEGs; this implies an image corrupted by Gaussian noise is also slightly corrupted by JPEG compression. Our benchmark tests networks with IMAGENET-C images, but networks should not be trained on these images. Networks should be trained on datasets such as ImageNet and not be trained on IMAGENET-C corruptions. To enable further experimentation, we designed an extra corruption type for each corruption category (Appendix B), and we provide CIFAR-10-C, TINY IMAGENET-C, IMAGENET 64 × 64-C, and Inception-sized editions. Overall, the IMAGENET-C dataset consists of 75 corruptions, all applied to ImageNet validation images for testing a pre-existing network. IMAGENET-P Design. The second benchmark that we propose tests the classifier's perturbation robustness. Models lacking in perturbation robustness produce erratic predictions which undermines user trust. When perturbations have a high propensity to change the model's response, then perturbations could also misdirect or destabilize iterative image optimization procedures appearing in style transfer BID14, decision explanations BID13, feature visualization BID46, and so on. Like IMAGENET-C, IMAGENET-P consists of noise, blur, weather, and digital distortions. Also as before, the dataset has validation perturbations; has difficulty levels; has CIFAR-10, Tiny ImageNet, ImageNet 64 × 64, standard, and Inception-sized editions; and has been designed for benchmarking not training networks. IMAGENET-P departs from IMAGENET-C by having perturbation sequences generated from each ImageNet validation image; examples are in FIG0. Each sequence contains more than 30 frames, so we counteract an increase in dataset size and evaluation time by using only 10 common perturbations. Common Perturbations. Appearing more subtly than the corruption from IMAGENET-C, the Gaussian noise perturbation sequence begins with the clean ImageNet image. The following frames in the sequence consist in the same image but with minute Gaussian noise perturbations applied. This sequence design is similar for the shot noise perturbation sequence. However the remaining perturbation sequences have temporality, so that each frame of the sequence is a perturbation of the previous frame. Since each perturbation is small, repeated application of a perturbation does not bring the image far out-of-distribution. For example, an IMAGENET-P translation perturbation sequence shows a clean ImageNet image sliding from right to left one pixel at a time; with each perturbation of the pixel locations, the ing frame is still of high quality. The perturbation sequences with temporality are created with motion blur, zoom blur, snow, brightness, translate, rotate, tilt (viewpoint variation through minor 3D rotations), and scale perturbations. IMAGENET-C Metrics. Common corruptions such as Gaussian noise can be benign or destructive depending on their severity. In order to comprehensively evaluate a classifier's robustness to a given type of corruption, we score the classifier's performance across five corruption severity levels and aggregate these scores. The first evaluation step is to take a trained classifier f, which has not been trained on IMAGENET-C, and compute the clean dataset top-1 error rate. Denote this error rate E f clean. The second step is to test the classifier on each corruption type c at each level of severity s (1 ≤ s ≤ 5). This top-1 error is written E f s,c. Before we aggregate the classifier's performance across severities and corruption types, we will make error rates more comparable since different corruptions pose different levels of difficulty. For example, fog corruptions often obscure an object's class more than brightness corruptions. We adjust for the varying difficulties by dividing by AlexNet's errors, but any baseline will do (even a baseline with 100% error rates, corresponding to an average of CEs). This standardized aggregate performance measure is the Corruption Error, computed with the formula DISPLAYFORM0. This in the mean CE or mCE for short. We now introduce a more nuanced corruption robustness measure. Consider a classifier that withstands most corruptions, so that the gap between the mCE and the clean data error is minuscule. Contrast this with a classifier with a low clean error rate which has its error rate spike in the presence of corruptions; this corresponds to a large gap between the mCE and clean data error. It is possible that the former classifier has a larger mCE than the latter, despite the former degrading more gracefully in the presence of corruptions. The amount that the classifier declines on corrupted inputs is given by the formula Relative CE DISPLAYFORM0. Averaging these 15 Relative Corruption Errors in the Relative mCE. This measures the relative robustness or the performance degradation when encountering corruptions. IMAGENET-P Metrics. A straightforward approach to estimate E ε∼E [P (x,y)∼D (f (ε(x)) = f (x))] falls into place when using IMAGENET-P perturbation sequences. Let us denote m perturbation sequences with S = x DISPLAYFORM1 where each sequence is made with perturbation p. The "Flip Probability" of network f: X → {1, 2, . . ., 1000} on perturbation sequences S is DISPLAYFORM2 For noise perturbation sequences, which are not temporally related, DISPLAYFORM3 1. We can recast the FP formula for noise sequences as FP DISPLAYFORM4. As was done with the Corruption Error formula, we now standardize the Flip Probability by the sequence's difficulty for increased commensurability. We have, then, the "Flip Rate" FR DISPLAYFORM5 Averaging the Flip Rate across all perturbations yields the mean Flip Rate or mFR. We do not define a "relative mFR" since we did not find any natural formulation, nor do we directly use predicted class probabilities due to differences in model calibration BID19.When the top-5 predictions are relevant, perturbations should not cause the list of top-5 predictions to shuffle chaotically, nor should classes sporadically vanish from the list. We penalize top-5 inconsistency of this kind with a different measure. Let the ranked predictions of network f on x be the permutation τ (x) ∈ S 1000. Concretely, if "Toucan" has the label 97 in the output space and "Pelican" has the label 145, and if f on x predicts "Toucan" and "Pelican" to be the most and second-most likely classes, respectively, then τ (x) = 1 and τ (x) = 2. These permutations contain the top-5 predictions, so we use permutations to compare top-5 lists. To do this, we define DISPLAYFORM6. If the top-5 predictions represented within τ (x) and τ (x) are identical, then d(τ (x), τ (x)) = 0. More examples of d on several permutations are in Appendix C. Comparing the top-5 predictions across entire perturbation sequences in the unstandardized Top-5 Distance uT5D DISPLAYFORM7 For noise perturbation sequences, we have uT5D DISPLAYFORM8 Once the uT5D is standardized, we have the Top-5 Distance T5D counts of the scenery before them. Hence, we propose the following protocol. The image recognition network should be trained on the ImageNet training set and on whatever other training sets the investigator wishes to include. Researchers should clearly state whether they trained on these corruptions or perturbations; however, this training strategy is discouraged (see Section 2). We allow training with other distortions (e.g., uniform noise) and standard data augmentation (i.e., cropping, mirroring), even though cropping overlaps with translations. Then the ing trained model should be evaluated on IMAGENET-C or IMAGENET-P using the above metrics. Optionally, researchers can test with the separate set of validation corruptions and perturbations we provide for IMAGENET-C and IMAGENET-P. How robust are current methods, and has progress in computer vision been achieved at the expense of robustness? As seen in Figure 3, as architectures improve, so too does the mean Corruption Error (mCE). By this measure, architectures have become progressively more successful at generalizing to corrupted distributions. Note that models with similar clean error rates have fairly similar CEs, and in TAB3 there are no large shifts in a corruption type's CE. Consequently, it would seem that architectures have slowly and consistently improved their representations over time. However, it appears that corruption robustness improvements are mostly explained by accuracy improvements. Recall that the Relative mCE tracks a classifier's accuracy decline in the presence of corruptions. Figure 3 shows that the Relative mCEs of many subsequent models are worse than that of AlexNet BID36. Full are in Appendix D. In consequence, from AlexNet to ResNet, corruption robustness in itself has barely changed. Thus our "superhuman" classifiers are decidedly subhuman. On perturbed inputs, current classifiers are unexpectedly bad. For example, a ResNet-18 on Scale perturbation sequences have a 15.6% probability of flipping its top-1 prediction between adjacent frames (i.e., FP ResNet-18 Scale = 15.6%); the uT5DResNet-18 Scale is 3.6. More are in Appendix E. Clearly perturbations need not be adversarial to fool current classifiers. What is also surprising is that while VGGNets are worse than ResNets at generalizing to corrupted examples, on perturbed examples they can be just as robust or even more robust. Likewise, Batch Normalization made VGG-19 less robust to perturbations but more robust to corruptions. Yet this is not to suggest that there is a fundamental trade-off between corruption and perturbation robustness. In fact, both corruption and perturbation robustness can improve together, as we shall see later. Be aware that Appendix F contains many informative failures in robustness enhancement. Those experiments underscore the necessity in testing on a a diverse test set, the difficulty in cleansing corruptions from image, and the futility in expecting robustness gains from some "simpler" models. Histogram Equalization. Histogram equalization successfully standardizes speech data for robust speech recognition BID58 BID20. For images, we find that preprocessing with Contrast Limited Adaptive Histogram Equalization BID48 ) is quite effective. Unlike our image denoising attempt (Appendix F), CLAHE reduces the effect of some corruptions while not worsening performance on most others, thereby improving the mCE. We demonstrate CLAHE's net improvement by taking a pre-trained ResNet-50 and fine-tuning the whole model for five epochs on images processed with CLAHE. The ResNet-50 has a 23.87% error rate, but ResNet-50 with CLAHE has an error rate of 23.55%. On nearly all corruptions, CLAHE slightly decreases the Corruption Error. The ResNet-50 without CLAHE preprocessing has an mCE of 76.7%, while with CLAHE the ResNet-50's mCE decreases to 74.5%.Multiscale Networks. Multiscale architectures achieve greater corruption robustness by propagating features across scales at each layer rather than slowly gaining a global representation of the input as in typical convolutional neural networks. Some multiscale architectures are called Multigrid Networks BID34. Multigrid networks each have a pyramid of grids in each layer which enables the subsequent layer to operate across scales. Along similar lines, Multi-Scale Dense Networks (MSDNets) BID31 use information across scales. MSDNets bind network layers with DenseNet-like BID30 ) skip connections. These two different multiscale networks both enhance corruption robustness, but they do not provide any noticeable benefit in perturbation robustness. Now before comparing mCE values, we first note the Multigrid network has a 24.6% top-1 error rate, as does the MSDNet, while the ResNet-50 has a 23.9% top-1 error rate. On noisy inputs, Multigrid networks noticeably surpass ResNets and MSDNets, as shown in Figure 5. Since multiscale architectures have high-level representations processed in tandem with fine details, the architectures appear better equipped to suppress otherwise distracting pixel noise. When all corruptions are evaluated, ResNet-50 has an mCE of 76.7%, the MSDNet has an mCE of 73.6%, and the Multigrid network has an mCE of 73.3%.Feature Aggregating and Larger Networks. Some recent models enhance the ResNet architecture by increasing what is called feature aggregation. Of these, DenseNets and ResNeXts BID60 are most prominent. Each purports to have stronger representations than ResNets, and the evidence is largely a hard-won ImageNet error-rate downtick. Interestingly, the IMAGENET-C mCE clearly indicates that DenseNets and ResNeXts have superior representations. Accordingly, a switch from a ResNet-50 (23.9% top-1 error) to a DenseNet-121 (25.6% error) decreases the mCE from 76.7% to 73.4% (and the relative mCE from 105.0% to 92.8%). More starkly, switching from a ResNet-50 to a ResNeXt-50 (22.9% top-1) drops the mCE from 76.7% to 68.2% (relative mCE decreases from 105.0% to 88.6%). Corruption robustness are summarized in Figure 5. This shows that corruption robustness may be a better way to measure future progress in representation learning than the clean dataset top-1 error rate. Some of the greatest and simplest robustness gains sometimes emerge from making recent models more monolithic. Apparently more representations, more redundancy, and more capacity allow these massive models to operate more stably on corrupted inputs. We saw earlier that making models smaller does the opposite. Swapping a DenseNet-121 (25.6% top-1) with the larger DenseNet-161 (22.9% top-1) decreases the mCE from 73.4% to 66.4% (and the relative mCE from 92.8% to 84.6%). In a similar fashion, a ResNeXt-50 (22.9% top-1) is less robust than the a giant ResNeXt-101 (21.0% top-1). The mCEs are 68.2% and 62.2% respectively (and the relative mCEs are 88.6% and 80.1% respectively). Both model size and feature aggregation are summarized in Figure 6. Consequently, future models with even more depth, width, and feature aggregation may attain further corruption robustness. Feature aggregation and their larger counterparts similarly improve perturbation robustness. While a ResNet-50 has a 58.0% mFR and a 78.3% mT5D, a DenseNet-121 obtains a 56.4% mFR and 76.8% mT5D, and a ResNeXt-50 does even better with a 52.4% mFR and a 74.2% mT5D. Reflecting the corruption robustness findings further, the larger DenseNet-161 has a 46.9% mFR and 69.5% mT5D, while the ResNeXt-101 has a 43.2% mFR and 65.9% mT5D. Thus in two senses feature aggregating networks and their larger versions markedly enhance robustness. Stylized ImageNet. BID16 propose a novel data augmentation scheme where ImageNet images are stylized with style transfer. The intent is that classifiers trained on stylized images will rely less on textural cues for classification. When a ResNet-50 is trained on typical ImageNet images and stylized ImageNet images, the ing model has an mCE of 69.3%, down from 76.7%.Adversarial Logit Pairing. ALP is an adversarial example defense for large-scale image classifiers BID33. Like nearly all other adversarial defenses, ALP was bypassed and has unclear value as an adversarial defense going forward BID11 ), yet this is not a decisive reason dismiss it. ALP provides significant perturbation robustness even though it does not provide much adversarial perturbation robustness against all adversaries. Although ALP was designed to increase robustness to small gradient perturbations, it markedly improves robustness to all sorts of noise, blur, weather, and digital IMAGENET-P perturbations-methods generalizing this well is a rarity. In point of fact, a publicly available Tiny ImageNet ResNet-50 model fine-tuned with ALP has a 41% and 40% relative decrease in the mFP and mT5D on TINY IMAGENET-P, respectively. ALP's success in enhancing common perturbation robustness and its modest utility for adversarial perturbation robustness highlights that the interplay between these problems should be better understood. In this paper, we introduced what are to our knowledge the first comprehensive benchmarks for corruption and perturbation robustness. This was made possible by introducing two new datasets, IMAGENET-C and IMAGENET-P. The first of which showed that many years of architectural advancements corresponded to minuscule changes in relative corruption robustness. Therefore benchmarking and improving robustness deserves attention, especially as top-1 clean ImageNet accuracy nears its ceiling. We also saw that classifiers exhibit unexpected instability on simple perturbations. Thereafter we found that methods such as histogram equalization, multiscale architectures, and larger featureaggregating models improve corruption robustness. These larger models also improve perturbation robustness. However, we found that even greater perturbation robustness can come from an adversarial defense designed for adversarial ∞ perturbations, indicating a surprising interaction between adversarial and common perturbation robustness. In this work, we found several methods to increase robustness, introduced novel experiments and metrics, and created new datasets for the rigorous study of model robustness, a pressing necessity as models are unleashed into safety-critical real-world settings. Clean Severity = 1 Severity = 2 Severity = 3 Severity = 4 Severity = 5Figure 7: Impulse noise modestly to markedly corrupts a frog, showing our benchmark's varying severities. In Figure 7, we show the Impulse noise corruption type in five different severities. Clearly, IMAGENET-C corruptions can range from negligible to pulverizing. Because of this range, the benchmark comprehensively assesses each corruption type. Speckle Noise Gaussian Blur Spatter Saturate Directly fitting the types of IMAGENET-C corruptions should be avoided, as it would cause researchers to overestimate a model's robustness. Therefore, it is incumbent on us to simplify model validation. This is why we provide an additional form of corruption for each of the four general types. These are available for download at https://github.com/hendrycks/robustness. There is one corruption type for each noise, blur, weather, and digital category in the validation set. The first corruption type is speckle noise, an additive noise where the noise added to a pixel tends to be larger if the original pixel intensity is larger. Gaussian blur is a low-pass filter where a blurred pixel is a of a weighted average of its neighbors, and farther pixels have decreasing weight in this average. Spatter can occlude a lens in the form of rain or mud. Finally, saturate is common in edited images where images are made more or less colorful. See FIG3 for instances of each corruption type. For some readers, the following function may be opaque, DISPLAYFORM0 where σ = (τ (x)) −1 τ (x) and the empty sum is understood to be zero. A high-level view of d is that it computes the deviation between the top-5 predictions of two prediction lists. For simplicity we find the deviation between the identity and σ rather than τ (x) and τ (x). In consequence we can consider d Also, d ((2, 3, 4, 5, 6, . . ., 1)) = 5. Distinctly, d ((1, 2, 3, 5, 6, 4, 7, 8, . . .)) = 2. As a final example, d ((5, 4, 3, 2, 1, 6, 7, 8, 9, . . .)) = 12.It may be that we want perturbation robustness for all predictions, including classes with lesser relevance. In such cases, it is still common that the displacement of the top prediction matters more than the displacement of, say, the 500th ranked class. For this there are many possibilities, such as the measure d (σ) = 1000 i=1 w i |w i − w σ(i) | such that w i = 1/i. This uses a Zipfian assumption about the rankings of the classes: the first class is n times as relevant as the nth class. Other possibilities involve using logarithms rather than hyperbolic functions as in the discounted cumulative gain BID37. One could also use the class probabilities provided by the model (should they exist). However such a measure could make it difficult to compare models since some models tend to be more uncalibrated than others BID19.As progress is made on this task, researchers may be interested in perturbations which are more likely to cause unstable predictions. To accomplish that, researchers can simply compare a frame with the frame two frames ahead rather than just one frame ahead. We provide concrete code of this slight change in the metric at https://github.com/hendrycks/robustness. For nontemporal perturbation sequences, i.e., noise sequences, we provide sequences where the noise perturbation is larger. IMAGENET-C corruption relative robustness are in BID32. IMAGENET-P mFR values are in TAB7, and mT5D values are in Stability Training. Stability training is a technique to improve the robustness of deep networks BID61. The method's creators found that training on images corrupted with noise can lead to underfitting, so they instead propose minimizing the cross-entropy from the noisy image's softmax distribution to the softmax of the clean image. The authors evaluated performance on images with subtle differences and suggested that the method provides additional robustness to JPEG corruptions. We fine-tune a ResNet-50 with stability training for five epochs. For training with noisy images, we corrupt images with uniform noise, where the maximum and minimum of the uniform noise is tuned over {0.01, 0.05, 0.1}, and the stability weight is tuned over {0.01, 0.05, 0.1}. Across all noise strengths and stability weight combinations, the models with stability training tested on IMAGENET-C have a larger mCEs than the baseline ResNet-50's mCE. Even on unseen noise corruptions, stability training does not increase robustness. However, the perturbation robustness slightly improves. The best model according to the IMAGENET-P validation set has an mFR of 57%, while the original ResNet's mFR is 58%. An upshot of this failure is that benchmarking robustness-enhancing techniques requires a diverse test set. Image Denoising. An approach orthogonal to modifying model representations is to improve the inputs using image restoration techniques. Although general image restoration techniques are not yet mature, denoising restoration techniques are not. We thus attempt restore an image with the denoising technique called non-local means BID3. The amount of denoising applied is determined by the noise estimation technique of BID10. Therefore clean images receive should nearly no modifications from the restoration method, while noisy images should undergo considerable restoration. We found that denoising increased the mCE from 76.7% to 82.1%. A plausible account is that the non-local means algorithm striped the images of their subtle details even when images lacked noise, despite having the non-local means algorithm governed by the noise estimate. Therefore, the gains in noise robustness were wiped away by subtle blurs to images with other types of corruptions, showing that targeted image restoration can prove harmful for robustness.10-Crop Classification. Viewing an object at several different locations may give way to a more stable prediction. Having this intuition in mind, we perform 10-crop classification. 10-crop classification is executed by cropping all four corners and cropping the center of an image. These crops and their horizontal mirrors are processed through a network to produce 10 predicted class probability distributions. We average these distributions to compute the final prediction. Of course, a prediction informed by 10-crops rather than a single central crop is more accurate. Ideally, this revised prediction should be more robust too. However, the gains in mCE do not outpace the gains in accuracy on a ResNet-50. In all, 10-crop classification is a computationally expensive option which contributes to classification accuracy but not noticeably to robustness. Smaller Models. All else equal, "simpler" models often generalize better, and "simplicity" frequently translates to model size. Accordingly, smaller models may be more robust. We test this hypothesis with CondenseNets BID29. A CondenseNet attains its small size via sparse convolutions and pruned filter weights. An off-the-shelf CondenseNet (C = G = 4) obtains a 26.3% error rate and a 80.8% mCE. On the whole, this CondenseNet is slightly less robust than larger models of similar accuracy. Even more pruning and sparsification yields a CondenseNet (C = G = 8) with both deteriorated performance (28.9% error rate) and robustness (84.6% mCE). Here again robustness is worse than larger model robustness. Though models fashioned for mobile devices are smaller and in some sense simpler, this does not improve robustness. Another goal for machine learning is to learn the fundamental structure of categories. Broad categories, such as "bird," have many subtypes, such as "cardinal" or "bluejay." Humans can observe previously unseen bird species yet still know that they are birds. A test of learned fundamental structure beyond superficial features is subtype robustness. In subtype robustness we test generalization to unseen subtypes which share share essential characteristics of a broader type. We repurpose the ImageNet-22K dataset for a closer investigation into subtype robustness. Subtype Robustness. A natural image dataset with a hierarchical taxonomy and numerous types and subtypes is ImageNet-22K, an ImageNet-1K superset. In this subtype robustness experiment, we manually select 25 broad types from ImageNet-22K, listed in the next paragraph. Each broad type has many subtypes. We call a subtype "seen" if and only if it is in ImageNet-1K and a subtype of one of the 25 broad types. The subtype is "unseen" if and only if it is a subtype of the 25 broad types and is from ImageNet-22K but not ImageNet-1K. In this experiment, the correct classification decision for an image of a subtype is the broad type label. We take pre-trained ImageNet-1K classifiers which have not trained on unseen subtypes. Next we fine-tune the last layer of these pre-trained ImageNet-1K classifiers on seen subtypes so that they predict one of 25 broad types. Then, we test the accuracy on images of seen subtypes and on images of unseen subtypes. Accuracy on unseen subtypes is our measure of subtype robustness. Seen and unseen accuracies are shown in FIG5, while the ImageNet-1K classification accuracy before fine-tuning is on the horizontal axis. Despite only having 25 classes and having trained on millions of images, these classifiers demonstrate a subtype robustness performance gap that should be far less pronounced. We also observe that the architectures proposed so far hardly deviate from the trendline.

We propose ImageNet-C to measure classifier corruption robustness and ImageNet-P to measure perturbation robustness

Represent the short summary for finding the corresponding scientific article (long)

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: Represent the article for finding a single-sentence TLDR

Question: The paper explores a novel methodology in source code obfuscation through the application of text-based recurrent neural network network (RNN) encoder-decoder models in ciphertext generation and key generation. Sequence-to-sequence models are incorporated into the model architecture to generate obfuscated code, generate the deobfuscation key, and live execution. Quantitative benchmark comparison to existing obfuscation methods indicate significant improvement in stealth and execution cost for the proposed solution, and experiments regarding the model’s properties yield positive regarding its character variation, dissimilarity to the original codebase, and consistent length of obfuscated code. The field of code obfuscation has aimed to tackle reverse-engineering of code bases for years. The entire basis of this methodology is that if a program is constructed with logic not easily recognizable by a reader, the logic would be preserved intact and the software would be intrinsically protected. Traditional tactics include creative uses of whitespace, redundant logical operations, unnecessary conditional operations, amongst others. The common issue with obfuscation is that it can be reverseengineered, the only factor for a malicious actor would be the amount of time needed to discover the logic. DeepObfusCode is a proposed methodology to use neural networks to convert the plaintext source code into a cipher text by using the propagating architecture of neural networks to compound the randomness factor in the creation of the ciphertext. Yet at the same time, neural networks have the ability to learn statistical patterns and generate weights to convert one text to another, in our case from the ciphertext to plaintext. This would eventually permit users to simply load the ciphertext and the key to self-execute the program without foreign users viewing the inner workings of the code. From an academic standpoint, this methodology redirects obfuscation methodology towards complete obfuscation in contrary of incremental obfuscation, and suggests the usage and development of asymmetric key infrastructure in obfuscation. Beyond sequence-to-sequence network models, further obfuscation models could be built with greater degree of resilience, and other deep learning methods could be harnessed to develop alternative techniques to obfuscate code. The methodology can be adopted for more efficient, more effective obfuscation for developers protecting their proprietary codebase or cloud computing services wishing to guarantee confidentiality to customers. The algorithmic architecture could be further incorporated into larger frameworks or infrastructures to render homomorphic encryption and ensure complete anonymity of the codebase during execution from the execution provider. The objective of code obfuscation is to mask the computational processes and logic behind software code, to protect trade secrets, intellectual property, or confidential data. As such, there have been eight generalized methods behind code obfuscation, namely: (i) Name obfuscation; (ii) Data obfuscation; (iii) Code flow obfuscation; (iv) Incremental obfuscation; (v) Intermediate code optimization; (vi) Debug information obfuscation; (vi) Watermarking; (vii) Source code obfuscation. Source code obfuscation, the focus of the paper, is the process that hides the meaning behind the source code, in the case that a third party obtains the code that the code renders itself un-understandable. Under each branch of obfuscation method, there are sub-techniques to reduce understandability of code that are shared by the other obfuscation methods, including control ordering (changing the execution order of program statements), control computation (changing the control flow of the program, such as inserting dead code to increase code complexity), data aggregation (changing the access to data structures after their conversion into other types), renamed identifiers (replacing identifiers with meaningless strings as new identifiers to reduce readability and comprehension of what certain functions or methods do). While existing methods tend to require manual altering of source code with the aforementioned methods, the proposed method performs complete obfuscation with relatively randomly generated characters as the code output. Malicious attackers or readers of the code will not be able to reverse-engineer the code based on the readability of the obfuscated code, and a well-lodged key file of the model weights (e.g. kept on the execution server) would prohibit de-obfuscation. Existing methods of evaluating obfuscation techniques have tended towards qualitative surveys of difficulty and time taken to de-obfuscate by students, software engineers or computer scientists. To quantitatively compare the performance of the proposed source code obfuscation method against existing code obfuscation methods, we will modify a framework that has been used to compare obfuscation methods before, but also malleable enough for us to adapt for our specific comparison use case. The four original comparison metrics are: (i) Code potency: This metric arbitrarily computes the degree of obfuscation with traditional complexity measures, with specific focus on control flow and data obfuscation. An example of an implementation would be frequency counts of misleading statements. (ii) Resilience: This metric measures the ability of the obfuscated text to withstand attacks from automated tools. (iii) Stealth: This metric tests the difficulty in manual de-obfuscation by humans. It inherently checks how quickly adversarial de-obfuscators can detect misdirecting statements, correctly interpret supposedly-confusing identifiers, and tests their ability to uncover the logic and process behind the code without prior or minimal knowledge of the program. (iv) Execution cost: This metric measures (i) the incremental time required to perform the obfuscation, and (ii) the incremental time required to execute the obfuscated code. This would be contrasted to the time taken for the original code without obfuscation. Since the proposed obfuscation method generates a ciphertext completely different from the original source text and cannot be reverse-engineered by manual human de-obfuscation, the former two metrics (code potency and resilience) would not be used as comparison metrics for this method. The main metrics for evaluating DeepObfusCode would be stealth and execution time. Neural networks have had recent applications in the field of cryptography and stenography. There has been a recent implementation of style transfer in stenography, where secret images are encrypted within a cover image, and the secret could be obtained by passing the encrypted image through a secondary network. There is another implementation of n-dimensional convolutional neural networks used to encrypt input data. Another implementation involves the use of a multilayer perception (MLP) model to encrypt satellite images, and decrypting using a secondary MLP model. Implementations on neural networks that can execute on encrpyted data has also been a recent development. This indicates a growing interest in encrypting data through the use of neural networks. Unlike prior on data encryption through deep learning techniques, the proposed architecture opens an alternative application, which is to encrypt source code itself through the use of deep learning, then using the generated model file to decrypt and execute the code. RNN Encoder-Decoders or sequence-to-sequence models, trained to predict an output text set from provided input text, have had growing applications in statistical machine translation, grammatical error correction, and text summarization. The notion of taking input sentences and converting them to labeled output sentences with a trained model of weights for character-by-character prediction would be the basis for the proposed obfuscation method. The proposed obfuscation utilizes the advantage of text-based encoder-decoders in character generation and calculating weights to convert one string into another. The architecture behind the obfuscation is first a primary recurrent neural network (RNN) encoderdecoder model with randomly-set weights taking the original code text as an input to generate the obfuscated text. Then a secondary RNN encoder-decoder model is passed two arguments, the generated ciphertext and the original code as the corresponding label, and is trained for a number of iterations to calculate weights to generate the original text, and the weights of the network would be the key generated. This section will discuss the architecture and implementation in further detail. Figure 1: Overview of ciphertext generation. We pass the source code and a character set as inputs to initialize both character sets for the encoder-decoder model, then randomly assign weights to generate the ciphertext, an obfuscated version of the source code. To generate the obfuscated code, we first take the original legible text (source code) and a full character set (a string containing all characters, including letters, numbers and punctuation). We perform text processing for both text strings to prepare the inputs for the encoder and decoder models. First we create unique character sets for both strings, then create two dictionaries for both strings to index each character (key is index and value is character), and create two other dictionaries for both strings to return character given its index (key is character and value is index). Finally we vectorize both strings. We use the source code character set as inputs for the encoder model and both text character sets for the decoder model. We will randomly generate weights for a given number of times n (the randomness index; in our experiment we set n = 10), and set the model weights to be the generated weights array. The variation in the weights array will determine the degree of randomness in the ciphertext. Weights generated will alter the character value generated for each segment of the string, as shown in the Ciphertext Generation function C(p). c refers to the ciphertext (obfuscated code), p refers to the plaintext (source code), N refers to each weight position. f n and w rand are the n-th feature and randomized weight, respectively. Z(p) is a normalization constant that is independent of the weights. The output will be the output sequence, to which we will decode by taking reference from the index-to-character dictionary to convert the output sequence into an output character string and iteratively append characters. The ing output would be the ciphertext. After generating the ciphertext, we use it along with the original source code plaintext to generate a key. We apply the same text processing steps as before, set ciphertext as input into the encoder, and both texts as inputs into the decoder, and train the model for a certain number of iteration counts. For our experiments, we tended to use 2000 iterations, though Early Stopping mechanisms could be used to lower training time for shorter source code; the iteration count should be the number of iterations needed to ensure the output text is executable and identical to the source code text. After the training and model weight setting process is complete, export the encoder and decoder model files in HDF5 format (the key), and export the metadata (the dictionaries of index to char, and char to index) in pickle format. The Key Generation function K(p, c) accepts the arguments c ciphertext and p plaintext, and sets weights while minimizing the loss function. f n and w n are the n-th feature and weight (after training), respectively. Z(c) is a normalization constant that is independent of the weights. w n f n (p|c) + log Z(c) After the ciphertext is decoded from the output sequence, we test if the code is executable; if it fails, we retrain the model (a pre-determined loss threshold that ensures correct execution would facilitate early stopping in iterative training); if it passes, the ciphertext and key pair are retained. During live execution, we have three inputs: the obfuscated code (ciphertext), the key (model files), and the metadata files; function K(c, k) would be modified to accept c ciphertext, and k model file (along with metadata file). For our own experiments, the model and metadata files were separate; for Figure 3: Overview of live execution. To run the obfuscated code on any server or system, one would pass in the obfuscated code into an execution engine that takes the ciphertext and the lodged model files as inputs to execute the withheld code. execution in live systems, it would be possible to combine them into a single file if preferred. When we pass all three through the model container, the output value is executed as soon as it is returned, i.e. Exec(K(c, k)). The evaluate the obfuscation method compared to existing obfuscation techniques, the method will be tested on two aspects, stealth and execution cost. Other parameters such as code potency or resilience would not be applicable to this method, as those comparison metrics require some form of the original code to be preserved; but since this obfuscation method completely regenerates the code base to be indistinguishable from the original code, such testing (e.g. searching for misleading control flow) will not work. The objective of testing for the stealthiness of obfuscated code is to measure the complexity or distance in randomness from the original code. Hence the logic behind our comparison is to first obtain recognized well-obfuscated code and its deobfuscated copy (the original code), obfuscate the original code using our proposed method, then compare the distance in similarity between the original code and each of the obfuscated code. Obfuscated code samples were obtained through the International Obfuscated C Code Contest (IOCCC), and de-obfuscated code samples of the selected obfuscated code was found on forums. The obfuscated samples were winners in the IOCCC contest, signifying high quality, and were manually created by programmers. A total of five obfuscated-deobfuscated paired samples were used for the experiment, which can be found in https://github.com/dattasiddhartha-1/dsource. Obfuscated code of reputable, unbiased quality and its deobfuscated counterpart is generally difficult to obtain in large samples; the aforementioned samples would be used in bench-marking and comparison, but not for testing the inherent properties of the proposed architecture. To test the obfuscation method extensively, for each de-obfuscated sample, we performed the obfuscation to obtain the ciphertext for 100 iterations (to return 500 samples of obfuscated code samples in total, 100 per de-obfuscated code sample). Then we used Levenshtein Distance to measure the distance between two code samples; the distance is the number of deletions, insertions, or substitutions required to transform the source string into the target string. After calculating the Levenshtein Distance values between the original de-obfuscated code and the IOCCC obfuscated code, we calculated the Levenshtein Distance values between the original de-obfuscated code with each respective sample's DeepObfusCode obfuscated code and took the mean Levenshtein Distance for the sample. The are tabulated in Table 1. From the table, we can observe that in general the DeepObfusCode obfuscation has a greater degree of randomness or dissimilarity from the original source text compared to the benchmark IOCCC obfuscated text, with an average magnitude improvement of 1.2614 (the average of the ratio of proposed:benchmark across all samples) and with a standard deviation 0.7176. Set 2 of the experiment indicates the benchmark outperforming the proposed method. Inspection of set 2 reveals a greater proportion of repetitive characters as junk code insertions, which serve to confuse de-obfuscators, but also dilute the similarity (conversely inflate the Levenstein distance) since the proportion of overlapping characters to the total number of characters would be lower. While the total collection indicates a general improvement in dissimilarity, removing set 2 would indicate an average magnitude improvement of 1.5287 and a standard deviation of 0.5352. This aids in justifying that the proposed obfuscation method performs well in the aspect of stealth. The primary focus of the section would be to measure how ciphertext generation time (time to encrypt) and key generation time (time to decrypt) would vary with the length of the source code (plaintext length). The experimental design starts with the random generation of strings varying from length 1 to 4000 (to vary the plaintext length). Then we set loggers to record information regarding the six main variables we would like to test: (i) Length of string input, (ii) Randomness metric (Levenshtein distance), (iii) Execution time for encryption, (iv) Execution time for decryption, (v) Character variation, (vi) Average character length of ciphertext. As execution time depends on the device running the simulation, for reference, the simulation was run on a Python Jupyter notebook, running on Windows 10 with a Nvidia Geforce 1070 GTX graphics card (running at 10% GPU usage level). The plot of ciphertext generation time requirement against length of the source code shows no distinct pattern, while the plot of key generation time requirement against length of the source code shows a linear pattern. This infers that ciphertext generation is not length-dependent and can be executed without significant incremental cost. Since the length of the source code affects the training time required for the same number of iterations (higher length would increase training time per epoch), longer source code would require more time to generate a key, so this obfuscation method may be more suitable for smaller code bases or systems with sufficient computing resources. We can also infer the key generation takes linear time, i.e. time complexity is O(n). Beyond execution cost, this experiment yielded additional information regarding the properties of this obfuscation model. Plotting ciphertext character variation and ciphertext length against plaintext length reveals: (i) the character variation is widely distributed regardless of the length of the plaintext input, which further supports the notion of randomness of ciphertext generation, as the ciphertext is based purely on the randomness in the model weight generation; (ii) the ciphertext length is kept low (on average 72 character length) regardless of the plaintext length, which reduces obfuscated code storage requirements. The notion that the cipher generation algorithm produces short random ciphertexts further implies it would be difficult for malicious actors to reverse-engineer the ciphertext by setting random weights or training a model without a known output text. For the model used in the experiment, the model file consists of 3 layers of arrays: the main layer contains 8 arrays, of which each array has an array-length as represented in, of which each sub-array (except the third and sixth array) contains an array of 1024 values. One would have to randomly generate 32-bit floats for 975,872 values, at least to a proximate range to the actual values before they can generate readable de-obfuscated code, but even then they would need to further generate values to the 8th decimal place if they intend to obtain scrambled text without any lapse in meaning. The correlation matrix in Figure 7 summarizes the relationships between the properties tested in the experiment. This paper presents a novel obfuscation methodology using RNN encoder-decoder models to generate ciphertext from source code and generating and utilizing model weights as keys. Figure 7: Correlation matrix of properties obfuscation methods, it is at least on par in terms of stealth and is expected to outperform for larger code bases in terms of obscurity and readability, and though key generation may take a significant amount of time for larger code bases or require more computational resources, it would be less time-intensive than to manually obfuscate the source code. This would be a good use case application for services that have confidential source code in plaintext but would prefer ciphertext yet require the ability to execute.

represent the short summary for finding the corresponding scientific article (long)

Obfuscate code using seq2seq networks, and execute using the obfuscated code and key pair

Represent the short summary for finding the corresponding scientific article (long)!

We use ideas from quantum computing to proposed word embeddings that utilize much fewer trainable parameters.


Question: represent the article for finding a single-sentence tldr

Question: We propose a method for joint image and per-pixel annotation synthesis with GAN. We demonstrate that GAN has good high-level representation of target data that can be easily projected to semantic segmentation masks. This method can be used to create a training dataset for teaching separate semantic segmentation network. Our experiments show that such segmentation network successfully generalizes on real data. Additionally, the method outperforms supervised training when the number of training samples is small, and works on variety of different scenes and classes. The source code of the proposed method will be publicly available.

Represent the short summary for finding the corresponding scientific article (long)

GAN-based method for joint image and per-pixel annotation synthesis

represent
To give you a sense - "General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities." should be close to "Various gradient compression schemes have been proposed to mitigate the communication cost in distributed training of large scale machine learning models. Sign-based methods, such as signSGD , have recently been gaining popularity because of their simple compression rule and connection to adaptive gradient methods, like ADAM. In this paper, we perform a general analysis of sign-based methods for non-convex optimization. Our analysis is built on intuitive bounds on success probabilities and does not rely on special noise distributions nor on the boundedness of the variance of stochastic gradients. Extending the theory to distributed setting within a parameter server framework, we assure exponentially fast variance reduction with respect to number of nodes, maintaining 1-bit compression in both directions and using small mini-batch sizes. We validate our theoretical findings experimentally. One of the key factors behind the success of modern machine learning models is the availability of large amounts of training data (; ;). However, the state-of-the-art deep learning models deployed in industry typically rely on datasets too large to fit the memory of a single computer, and hence the training data is typically split and stored across a number of compute nodes capable of working in parallel. Training such models then amounts to solving optimization problems of the form where f m: R d → R represents the non-convex loss of a deep learning model parameterized by x ∈ R d associated with data stored on node m. Arguably, stochastic gradient descent (SGD) (; ;) in of its many variants (; ; ; ;) is the most popular algorithm for solving. In its basic implementation, all workers m ∈ {1, 2, . . ., M} in parallel compute a random approximation g m (x k) of ∇f m (x k), known as the stochastic gradient. These approximations are then sent to a master node which performs the aggregation The aggregated vector is subsequently broadcast back to the nodes, each of which performs an update of the form x k+1 = x k − γ kĝ (x k), thus updating their local copies of the parameters of the model. Typically, communication of the local gradient estimatorsĝ m (x k) to the master forms the bottleneck of such a system (; ;). In an attempt to alleviate this communication bottleneck, a number of compression schemes for gradient updates have been proposed and analyzed;; signSGD, Theorem 1 Step size Weak noise assumptions? ρi > ). A compression scheme is a (possibly randomized) mapping Q: R d → R d, applied by the nodes toĝ m (x k) (and possibly also by the master to aggregated update in situations when broadcasting is expensive as well) in order to reduce the number of bits of the communicated message. Sign-based compression. Although most of the existing theory is limited to unbiased compression schemes, i.e., on operators Q satisfying EQ(x) = x, biased schemes such as those based on communicating signs of the update entries only often perform much better (; ; ; ; ; ; ;). The simplest among these sign-based methods is signSGD (see also Algorithm 1; Option 1), whose update direction is assembled from the component-wise signs of the stochastic gradient. Adaptive methods. While ADAM is one of the most popular adaptive optimization methods used in deep learning , there are issues with its convergence and generalization properties. It was noted in that the behaviour of ADAM is similar to a momentum version of signSGD. Connection between sign-based and adaptive methods has long history, originating at least in Rprop and RMSprop . Therefore, investigating the behavior of signSGD can improve our understanding on the convergence of adaptive methods such as ADAM. We now summarize the main contributions of this work. Our key are summarized in Table 1. 1 In fact, bounded variance assumption, being weaker than bounded second moment assumption, is stronger (or, to be strict, more curtain) than SPB assumption in the sense of differential entropy, but not in the direct sense. The entropy of probability distribution under the bounded variance assumption is bounded, while under the SPB assumption it could be arbitrarily large. This observation is followed by the fact that for continuous random variables, the Gaussian distribution has the maximum differential entropy for a given variance (see https://en.wikipedia.org/wiki/Differential_entropy). • 2 methods for 1-node setup. In the M = 1 case, we study two general classes of sign based methods for minimizing a smooth non-convex function f. The first method has the standard form while the second has a new form not considered in the literature before: • Key novelty. The key novelty of our methods is in a substantial relaxation of the requirements that need to be imposed on the gradient estimatorĝ(x k) of the true gradient ∇f (x k). In sharp contrast with existing approaches, we allowĝ(x k) to be biased. Remarkably, we only need one additional and rather weak assumption onĝ(x k) for the methods to provably converge: we require the signs of the entries ofĝ(x k) to be equal to the signs of the entries of ∇f (x k) with a probability strictly larger than 1 /2 (see Section 2; Assumption 1). We show through a counterexample (see Section 2.2) that this assumption is necessary. • Geometry. As a byproduct of our analysis, we uncover a mixed l 1 -l 2 geometry of sign descent methods (see Section 3). • Convergence theory. We perform a complexity analysis of methods and (see Section 4.1; Theorem 1). While our complexity bounds have the same O(1 / √ K) dependence on the number of iterations, they have a better dependence on the smoothness parameters associated with f. Theorem 1 is the first on signSGD for non-convex functions which does not rely on mini-batching, and which allows for step sizes independent of the total number of iterations K. Finally, Theorem 1 in can be recovered from our general Theorem 1. Our bounds are cast in terms of a novel norm-like function, which we call the ρ-norm, which is a weighted l 1 norm with positive variable weights. • Distributed setup. We extend our to the distributed setting with arbitrary M (Section 4.2), where we also consider sign-based compression of the aggregated gradients. In this section we describe our key (and weak) assumption on the gradient estimatorĝ(x) of the true gradient ∇f (x), and give an example which shows that without this assumption, method can fail. Assumption 1 (SPB: Success Probability Bounds). For any x ∈ R d, we have access to an independent (and not necessarily unbiased) estimatorĝ(x) of the true gradient g(x):= ∇f (x) that satisfies for all x ∈ R d and all i ∈ {1, 2, . . ., d}. We will refer to the probabilities ρ i as success probabilities. As we will see, they play a central role in the convergence of sign based methods. We stress that Assumption 1 is the only assumption on gradient noise in this paper. Moreover, we argue that it is reasonable to require from the sign of stochastic gradient to show true gradient direction more likely than the opposite one. Extreme cases of this assumption are the absence of gradient noise, in which case ρ i = 1, and an overly noisy stochastic gradient, in which case ρ i ≈ 1 2. Remark 1. Assumption 1 can be relaxed by replacing bounds with However, if Prob(signĝ i (x) = 0) = 0 (e.g. in the case ofĝ i (x) has continuous distributions), then these two bounds are identical. Under review as a conference paper at ICLR 2020 Extension to stochastic sign oracle. Notice that we do not requireĝ to be unbiased. Moreover, we do not assume uniform boundedness of the variance, or of the second moment. This observation allows to extend existing theory to more general sign-based methods with a stochastic sign oracle. By a stochastic sign oracle we mean an oracle that takes x k ∈ R d as an input, and outputs a random vectorŝ k ∈ R d with entries in ±1. However, for the sake of simplicity, in the rest of the paper we will work with the signSGD formulation, i.e., we letŝ k = signĝ(x k). Here we analyze a counterexample to signSGD discussed in. Consider the following least-squares problem with unique minimizer x * =:, where ε ∈ and stochastic gradientĝ(x) = ∇ a i, x 2 = 2 a i, x a i with probabilities 1/2 for i = 1, 2. Let us take any point from the line l = {(z 1, z 2): z 1 + z 2 = 2} as initial point x 0 for the algorithm and notice that signĝ(x) = ±(1, −1) for any x ∈ l. Therefore, signSGD with any step-size sequence remains stuck along the line l, whereas the problem has a unique minimizer at the origin. We now investigate the cause of the divergence. In this counterexample, Assumption 1 is violated. Indeed, note that for i = 1, 2. By S:= {x ∈ R 2 : a 1, x · a 2, x > 0} = ∅ denote the open cone of points having either an acute or an obtuse angle with both a i' s. Then for any x ∈ S, the sign of the stochastic gradient is ±(1, −1) with probabilities 1 /2. Hence for any x ∈ S, we have low success probabilities: So, in this case we have an entire conic region with low success probabilities, which clearly violates. Furthermore, if we take a point from the complement open coneS c, then the sign of stochastic gradient equals to the sign of gradient, which is perpendicular to the axis of S (thus in the next step of the iteration we get closer to S). For example, if a 1, x < 0 and a 2, x > 0, then signĝ(x) = (1, −1) with probability 1, in which case x − γ signĝ(x) gets closer to low success probability region S. In summary, in this counterexample there is a conic region where the sign of the stochastic gradient is useless (or behaves adversarially), and for any point outside that region, moving direction (which is the opposite of the sign of gradient) leads toward that conic region. To justify our SPB assumption, we show that it holds under general assumptions on gradient noise. Lemma 1 (see B.1). Assume that for any point x ∈ R d, we have access to an independent and unbiased estimatorĝ(x) of the true gradient g(x). Assume further that each coordinateĝ i has a unimodal and symmetric distribution with variance σ Next, we remove the distribution condition and add a strong growth condition together with fixed mini-batch size. Lemma 2 (see B.2). Assume that for any point x ∈ R d, we have access to an independent, unbiased estimatorĝ(x) of the true gradient g(x), with coordinate-wise bounded variances σ for some constant c. Then, choosing a mini-batch size τ > 2c, we get Finally, we give an adaptive condition on mini-batch size for the SPB assumption to hold. Lemma 3 (see B.3). Assume that for any point x ∈ R d we have access to an independent and unbiased estimatorĝ(x) of the true gradient g(x). Let σ 2 i = σ 2 i (x) be the variance and ν Under review as a conference paper at ICLR 2020 In this section we introduce the concept of a norm-like function, which call ρ-norm, induced from success probabilities. Used to measure gradients in our convergence rates, ρ-norm is a technical tool enabling the analysis. be the collection of probability functions from the SPB assumption. We define the ρ-norm of gradient Note that ρ-norm is not a norm as it may not satisfy the triangle inequality. However, under SPB assumption, ρ-norm is positive definite as it is a weighted l 1 norm with positive (and variable) weights, and g ρ = 0 if and only if g = 0. Under the assumptions of Lemma 2, ρ-norm can be lower bounded by a weighted l 1 norm with positive constant weights 1 − 2c Under the assumptions of Lemma 1, ρ-norm can be lower bounded by a mixture of the l 1 and squared l 2 norms: Note that l 1,2 -norm is again not a norm. However, it is positive definite, continuous and order preserving, i.e., for any g k, g,g ∈ R d we have: i) g l 1,2 ≥ 0 and g l 1,2 = 0 if and only if g = 0;. From these three properties it follows that g k l 1,2 → 0 implies g k → 0. These properties are important as we will measure convergence rate in terms of the l 1,2 norm in the case of unimodal and symmetric noise assumption. To understand the nature of the l 1,2 norm, consider the following two cases when σ i (x) ≤ c|g i (x)| +c for some constants c,c ≥ 0. If the iterations are in ε-neighbourhood of a minimizer x * with respect to the l ∞ norm (i.e., max 1≤i≤d |g i | ≤ ε), then the l 1,2 norm is equivalent to scaled l 2 norm squared: On the other hand, if iterations are away from a minimizer (i.e., min 1≤i≤d |g i | ≥ L), then the l 1,2 -norm is equivalent to scaled l 1 norm: g 1. These equivalences are visible in Figure 1, where we plot the level sets of g → g l 1,2 at various distances from the origin. Similar mixed norm observation was also noted in. Now we turn to our theoretical of sign based methods. First we give our general convergence under the SPB assumption. Afterwards, we present convergence in the distributed setting under the unimodal and symmetric noise assumptions. Throughout the paper we assume that f: and is L-smooth with some non-negative constants That is, we assume that We allow f to be nonconvex. We now state our convergence for Algorithm 1 under the general SPB assumption. Under review as a conference paper at ICLR 2020 Theorem 1 (Non-convex convergence of signSGD, see B.4). Under the SPB assumption, signSGD (Algorithm 1 with Option 1) with step sizes γ k = γ 0 / √ k + 1 converges as follows If γ k ≡ γ > 0, we get 1 /K convergence to a neighbourhood of the solution: We now comment on the above : • Generalization. Theorem 1 is the first general on signSGD for non-convex functions without mini-batching, and with step sizes independent of the total number of iterations K. Known convergence (; on signSGD use mini-batches and/or step sizes dependent on K. Moreover, they also use unbiasedness and unimodal symmetric noise assumptions, which are stronger assumptions than our SPB assumption (see Lemma 1). Finally, Theorem 1 in can be recovered from Theorem 1 (see Section D for the details). • Convergence rate. Rates and can be arbitrarily slow, depending on the probabilities ρ i. This is to be expected. At one extreme, if the gradient noise was completely random, i.e., if ρ i ≡ 1/2, then the ρ-norm would become identical zero for any gradient vector and rates would be trivial inequalities, leading to divergence as in the counterexample. At other extreme, if there was no gradient noise, i.e., if ρ i ≡ 1, then the ρ-norm would be just the l 1 norm and from we get the rateÕ(1/ √ K) with respect to the l 1 norm. However, if we know that ρ i > 1/2, then we can ensure that the method will eventually converge. • Geometry. The presence of the ρ-norm in these rates suggests that there is no particular geometry (e.g., l 1 or l 2) associated with signSGD. Instead, the geometry is induced from the success probabilities. For example, in the case of unbiased and unimodal symmetric noise, the geometry is described by the mixture norm l 1,2. • Practicality. The rate (as well as) supports the common learning schedule practice of using a constant step size for a period of time, and then halving the step-size and continuing this process. For a reader interested in comparing Theorem 1 with a standard for SGD, we state the standard in the Section C. We now state a general convergence rate for Algorithm 1 with Option 2. Theorem 2 (see B.5). Under the SPB assumption, Algorithm 1 (Option 2) with step sizes γ k = γ 0 / √ k + 1 converges as follows: In the case of constant step size γ k = γ > 0, the same rate as is achieved. Comparing Theorem 2 with Theorem 1, notice that a small modification in Algorithm 1 can remove the log-dependent factor from; we then bound the average of past gradient norms instead of the minimum. On the other hand, in a big data regime, function evaluations in Algorithm 1 (Option 2, line 4) are infeasible. Clearly, Option 2 is useful only when one can afford function evaluations and has rough estimates about the gradients (i.e., signs of stochastic gradients). This option should be considered within the framework of derivative-free optimization. In this part we present the convergence of distributed signSGD (Algorithm 2) with majority vote introduced in. Majority vote is considered within a parameter server framework, where for each coordinate parameter server receives one sign from each node and sends Under review as a conference paper at ICLR 2020 back the sign sent by the majority of nodes. Known convergence (; use O(K) mini-batch size as well as O(1/K) constant step size. In the sequel we remove this limitations extending Theorem 1 to distributed training. In distributed setting the number of nodes M get involved in geometry introducing new ρ M -norm, which is defined by the regularized incomplete beta function I (see B.6). Now we can state the convergence rate of distributed signSGD with majority vote. Theorem 3 (Non-convex convergence of distributed signSGD, see B.6). Under SPB assumption, distributed signSGD (Algorithm 2) with step sizes γ k = γ 0 / √ k + 1 converges as follows For constant step sizes γ k ≡ γ > 0, we have convergence up to a level proportional to step size γ: Variance Reduction. Using Hoeffding's inequality, we show that 1, where ρ(x) = min 1≤i≤d ρ i (x) > 1 /2. Hence, in some sense, we have exponential variance reduction in terms of number of nodes (see B.7). Number of nodes. Notice that theoretically there is no difference between 2l−1 and 2l nodes, and this in not a limitation of the analysis. Indeed, as it is shown in the proof, expected sign vector at the master with M = 2l − 1 nodes is the same as with M = 2l nodes: E sign(ĝ is the sum of stochastic sign vectors aggregated from nodes. The intuition behind this phenomenon is that majority vote with even number of nodes, e.g. M = 2l, fails to provide any sign Under review as a conference paper at ICLR 2020 with little probability (it is the probability of half nodes voting for +1, and half nodes voting for −1). However, if we remove one node, e.g. M = 2l − 1, then master receives one sign-vote less but gets rid of that little probability of failing the vote (sum of odd number of ±1 cannot vanish). So, somehow this two things cancel each other and we gain no improvement in expectation adding one more node to parameter server framework with odd number of nodes. We verify our theoretical experimentally using the MNIST dataset with feed-forward neural network (FNN) and the well known Rosenbrock (non-convex) function with d = 10 variables: Stochastic formulation of minimization problem for Rosenbrock function is as follows: at any point x ∈ R d we have access to biased stochastic gradientĝ(x) = ∇f i (x) + ξ, where index i is chosen uniformly at random from {1, 2, . . ., d − 1} and ξ ∼ N (0, ν 2 I) with ν > 0. Figure 2 illustrates the effect of multiple nodes in distributed training with majority vote. As we see increasing the number of nodes improves the convergence rate. It also supports the claim that in expectation there is no improvement from 2l − 1 nodes to 2l nodes. Figure 4 shows the robustness of SPB assumption in the convergence rate with constant step size. We exploited four levels of noise in each column to demonstrate the correlation between success probabilities and convergence rate. In the first experiment (first column) SPB assumption is violated strongly and the corresponding rate shows divergence. In the second column, probabilities still violating SPB assumption are close to the threshold and the rate shows oscillations. Next columns show the improvement in rates when success probabilities are pushed to be close to 1. Under review as a conference paper at ICLR 2020 Figure 5: Performance of signSGD with variable step size (γ 0 = 0.25) under four different noise levels (mini-batch size 1, 2, 5, 7) using Rosenbrock function. As in the experiments of Figure 4 with constant step size, these plots show the relationship between success probabilities and the convergence rate. In low success probability regime (first and second columns) we observe oscillations, while in high success probability regime (third and forth columns) oscillations are mitigated substantially. Under review as a conference paper at ICLR 2020 to a neighborhood of the solution. We fixed gradient noise level by setting mini-batch size 2 and altered the constant step size. For the first column we set bigger step size γ = 0.25 to detect the divergence (as we slightly violated SPB assumption). Then for the second and third columns we set γ = 0.1 and γ = 0.05 to expose the convergence to a neighborhood of the minimizer. For the forth column we set even smaller step size γ = 0.01 to observe a slower convergence. Here we state the well-known Gauss's inequality on unimodal distributions 3. Theorem 4 (Gauss's inequality). Let X be a unimodal random variable with mode m, and let σ 2 m be the expected value of (X − m) 2. Then for any positive value of r, Applying this inequality on unimodal and symmetric distributions, direct algebraic manipulations give the following bound: where m = µ and σ 2 m = σ 2 are the mean and variance of unimodal, symmetric random variable X, and r ≥ 0. Now, using the assumption that eachĝ i (x) has unimodal and symmetric distribution, we apply this bound for and get a bound for success probabilities Improvment on Lemma 1 and l 1,2 norm: The bound after Gauss inequality can be improved including a second order term Hence, continuing the proof of Lemma 1, we get and we could have defined l 1,2 -norm in a bit more complicated form as Under review as a conference paper at ICLR 2020 B.2 SUFFICIENT CONDITIONS FOR SPB: PROOF OF LEMMA 2 Letĝ (τ) be the gradient estimator with mini-batch size τ. It is known that the variance forĝ (τ) is dropped by at least a factor of τ, i.e. Hence, estimating the failure probabilities of signĝ (τ) when g i = 0, we have which imples We will split the derivation into three lemmas providing some intuition on the way. The first two lemmas establish success probability bounds in terms of mini-batch size. Essentially, we present two methods: one works well in the case of small randomness, while the other one in the case of non-small randomness. In the third lemma, we combine those two bounds to get the condition on mini-batch size ensuring SPB assumption. Lemma 4. Let X 1, X 2,..., X τ be i.i.d. random variables with non-zero mean µ:= EX 1 = 0, finite variance σ 2:= E|X 1 − µ| 2 < ∞. Then for any mini-batch size τ ≥ 1 Proof. Without loss of generality, we assume µ > 0. Then, after some adjustments, the proof follows from the Chebyshev's inequality: where in the last step we used independence of random variables X 1, X 2,..., X τ. Obviously, bound is not optimal for big variance as it becomes a trivial inequality. In the case of non-small randomness a better bound is achievable additionally assuming the finitness of 3th central moment. Lemma 5. Let X 1, X 2,..., X τ be i.i.d. random variables with non-zero mean µ:= EX 1 = 0, positive variance σ 2:= E|X 1 − µ| 2 > 0 and finite 3th central moment ν 3:= E|X 1 − µ| 3 < ∞. Then for any mini-batch size τ ≥ 1 where error function erf is defined as Proof. Again, without loss of generality, we may assume that µ > 0. Informally, the proof goes as follows. As we have an average of i.i.d. random variables, we approximate it (in the sense of distribution) by normal distribution using the Central Limit Theorem (CLT). Then we compute success probabilities for normal distribution with the error function erf. Finally, we take into account the approximation error in CLT, from which the third term with negative sign appears. More formally, we apply Berry-Esseen inequality 4 on the rate of approximation in CLT : where N ∼ N has the standard normal distribution. Setting t = −µ √ τ /σ, we get It remains to compute the second probability using the cumulative distribution function of normal distribuition and express it in terms of the error function: Clearly, bound is better than when randomness is high. On the other hand, bound is not optimal for small randomness (σ ≈ 0). Indeed, one can show that in a small randomness regime, while both variance σ 2 and third moment ν 3 are small, the ration ν/σ might blow up to infinity producing trivial inequality. For instance, taking X i ∼ Bernoulli(p) and letting p → 1 gives ν/σ = O (1 − p) − 1 /6. This behaviour stems from the fact that we are using CLT: less randomness implies slower rate of approximation in CLT. As a of these two bounds on success probabilities, we conclude a condition on mini-batch size for the SPB assumption to hold. Under review as a conference paper at ICLR 2020 Lemma 6. Let X 1, X 2,..., X τ be i.i.d. random variables with non-zero mean µ = 0 and finite variance σ 2 < ∞. Then where ν 3 is (possibly infinite) 3th central moment. Proof. First, if σ = 0 then the lemma holds trivially. If ν = ∞, then it follows immediately from Lemma 4. Assume both σ and ν are positive and finite. In case of τ > 2σ 2 /µ 2 we apply Lemma 4 again. Consider the case τ ≤ 2σ 2 /µ 2, which implies we get which together with gives Hence, SPB assumption holds if It remains to show that erf Lemma follows from Lemma applying it to i.i.d. dataĝ First, from L-smoothness assumption we have where g k = g(x k),ĝ k =ĝ(x k),ĝ k,i is the i-th component ofĝ k andL is the average value of L i' s. Taking conditional expectation given current iteration x k gives Under review as a conference paper at ICLR 2020 Using the definition of success probabilities ρ i we get Plugging this into and taking full expectation, we get Therefore Now, in case of decreasing step sizes where we have used the following standard inequalities In the case of constant step size γ k = γ Under review as a conference paper at ICLR 2020 B.5 CONVERGENCE ANALYSIS: PROOF OF THEOREM 2 Clearly, the iterations {x k} k≥0 of Algorithm 1 (Option 2) do not increase the function value in any iteration, i.e. E[f (x k+1)|x k ] ≤ f (x k). Continuing the proof of Theorem 1 from, we get where we have used the following inequality The proof for constant step size is the same as in Theorem 1. The proof of Theorem 3 goes with the same steps as in Theorem 1, except the derivation- is replaced by where we have used the following lemma. Lemma 7. Assume that for some point x ∈ R d and some coordinate i ∈ {1, 2, . . ., d}, master node receives M independent stochastic signs signĝ m i (x), m = 1,..., M of true gradient g i (x) = 0. Let g (M) (x) be the sum of stochastic signs aggregated from nodes: where l = [(M +1) /2] and ρ i > 1 /2 is the success probablity for coordinate i."

We propose a novel adversarial training with domain adaptation method that significantly improves the generalization ability on adversarial examples from different attacks.