Question:
represent the scientific article to find a short tldr
Examples:
'Pre-trained word embeddings are the primary method for transfer learning in several Natural Language Processing (NLP) tasks. Recent works have focused on using unsupervised techniques such as language modeling to obtain these embeddings. In contrast, this work focuses on extracting representations from multiple pre-trained supervised models, which enriches word embeddings with task and domain specific knowledge. Experiments performed in cross-task, cross-domain and crosslingual settings indicate that such supervised embeddings are helpful, especially in the lowresource setting, but the extent of gains is dependent on the nature of the task and domain. Named entity recognition, semantic role labelling, relation extraction etc. can be thought of as primary tasks necessary for solving high level tasks like question answering, summarization etc. However, labelling large amounts of data at this granularity is not only prohibitively expensive, but also unscalable. Given that high performance models for these tasks already exist, it is desirable to leverage them for other language understanding tasks. Next, consider the domain adaptation setting where some domains have a lot of data, while others do not. A model for a low-resource domain would benefit from information in expert models trained on other data rich domains. Finally, consider the setting of cross-lingual adaptation, a common problem for personal assistants expanding to more languages. As the number of languages increases, it becomes unfeasible to obtain human annotated data. Again, the need to adapt to low-resource languages can be met by leveraging models that already exist for high-resource languages. Motivated by the above scenarios, we propose a simple method to transfer supervised knowledge, from multiple sources, in an easy to implement manner. In our approach, this knowledge is extracted from source models in the form of contextual word embeddings. We treat preexisting models as embedding extractors, which are used to extract token level representations for an input sentence. These representations are then combined via a task specific convex combination. Unsupervised transfer learning methods such as ELMo have shown great success for a variety of tasks BID15. While they have the advantage of being trained on very large corpora, the training objectives are unsupervised. We show that in low-resource settings especially, leveraging representations from multiple pre-trained supervised models in related tasks, domains or languages can prove to be beneficial. The common way of supervised transfer learning via fine-tuning can transfer information only from a single source task BID11. One way to incorporate information from multiple external sources is via multi-task learning BID5 BID17. The limitations of multitask learning are the need for labelled data for the source models, longer training times and complex design decisions (weighing the losses for each task, sampling strategies, and choice of architecture). In contrast, our plug-and-play approach is simple and does not assume availability of source model data at training time. Finally, our approach also provides some interpretability (through the parameters of the convex combination) into which source tasks or domains are important for which other tasks and domains. Our work aligns most with the following three directions of research. Unsupervised transfer learning Embeddings such as GloVe and FastText have become an integral part of the modern NLP pipeline BID13; BID0. Over the last year, language model based deep contextualized embedding methods such as ELMo have shown substantial improvements over their shallow counterparts, heralding a new era of word representations BID15. In terms of modelling approach, our work is similar to BID7, where the authors use multiple existing models for domain adaptation for spoken language understanding. In comparison, our work focuses not just on the domain adaptation, but also the cross-task and cross-lingual settings. In another work, BID1 create metaembeddings from multiple embeddings like GloVe, Fasttext etc. Most deep learning models can be thought of as having an encoder E and decoder D. For example in a Deep-SRL model BID6, stacked bidirectional LSTM constitutes E, while D is the softmax layer. Assume K existing supervised models either for different tasks or different domains M 1,..., M K and corresponding encoders E 1,..., E K. Given a sentence of N tokens (t 1, t 2, ..., t N), we feed these tokens to the K different encoders and get K different representations for each token. We denote the encoder output of the kth model for the nth token by h k n. Each encoder generates representations specialized for the task, domain, or language it was trained for. Since our approach assumes no explicit information about the encoders of the model, they can be of varying dimensions and use different underlying architectures. Evidently, they would also be in different vector spaces and therefore we first use a projection layer to bring all of them in the same vector space. The parameters of these projection layers W 1,... W K are learned along with the target model parameters. DISPLAYFORM0 For inclusion in a downstream model, we aggregate the projection layer output of all the different source models into one vector. Several aggregation schemes can be employed: pooling, convex combination, attention etc. We choose the simple yet interpretable convex combination approach, as described below. This technique is similar to one used by ELMo BID15. We use a softmax normalized weight s k corresponding to each of the different representations of the word, add them up and use a scalar parameter γ that scales up the whole vector. The embedding O n for the nth word comes out to be: DISPLAYFORM0 This approach adds K + 1 trainable parameters to the model. An advantage of combining the representations in this manner is that the size of the embedding is fixed irrespective of the number of source models used. Once we get a combined representation, it can be used in the target model just like any other embedding. In our experiments, we concatenate these embeddings with traditional GloVe or ELMo embeddings. We use the proposed supervised contextual embeddings along with GloVe and ELMo embeddings in three knowledge transfer settings. Cross-task transfer In this setting, we transfer knowledge to a target task from models trained on multiple source tasks. We transfer into Semantic Role Labeling (SRL) task using Constituency Parsing (CP), Dependency Parsing (DP) and Named Entity Recognition (NER) as source tasks. The choice of SRL as a target task, with source embeddings from CP, DP and NER models, is inspired by the popular use of explicit syntactic parsing features for SRL. We use OntoNotes 5.0 BID16 dataset to train the SRL target tasks. We use the stacked alternating LSTM architechture for SRL as per BID6. On the source side, the DP model is based on BID4 and CP on BID18. For most of the source models, we use off-the-shelf, pre-trained models provided by AllenNLP 1. We refer readers to BID15 for further description of model architectures for the various tasks. Cross-domain transfer Here, we study the applicability of our method in the cross-domain setting. The target task is same as the source tasks, but instead, the domains of the source and target models are different. For this set of experiments, our task is NER and we use the OntoNotes 5.0 dataset which comes with annotations for multiple domains. Though NER is an easier task, we chose it as the target task for the cross-domain setting as even state of the art NER models may perform poorly for a data-scarce domain. We choose the target domain as web blogs and the source domains are newswire, broadcast conversation, telephone conversation, magazines and broadcast news. Note that the samples in the validation and test sets are also limited to the web blogs domain only. We use an LSTM-CRF architechture with 1 LSTM layer for NER as per BID14.Cross-lingual transfer From the CoNLL shared tasks, we obtain NER datasets for English, Spanish, German and Dutch Tjong BID20. We consider two scenarios with German and Spanish as the target languages and the remaining 3 as source languages. To facilitate the input of sentences into models from other languages with different scripts, we rely on crosslingual embeddings provided by MUSE Conneau et al. (2017b). The NER model architecture is the same as the one used for the cross-domain experiments. To study the effectiveness of our approach in the low resource setting, in addition to the full datasets, we also run experiments on smaller training subsets. Similar to BID12, we create random subsets of 1,000 and 5,000 samples to simulate a low resource setting. In all the aforementoiend settings, the source task models are trained on their complete datasets. Hyperparameters We use the Adam optimizer (lr=0.001) for all our experiments. We run our target models for 50 epochs in SRL tasks and 75 epochs for NER tasks. Batch size is kept at 8 for the 1k data setting and 16 for 5k data setting. The dimensions of the GloVe and ELMo embeddings are 100 and 1024 respectively. The output dimension of the projection layer in all settings for supervised embeddings is 300. Cross-task SRL (with GloVe and ELMo in 1k, 5k and full data settings) have been tabulated in TAB1 has the for cross-domain NER and TAB2 shows the for crosslingual transfer on NER. All the reported numbers are F1 scores. Cross-task SRL With GloVe embeddings, adding the supervised embeddings gives us significant improvements in F1 scores ∼ 5% for 1k and ∼ 7% for 5k examples. When we use the entire dataset, adding supervised embeddings provides no performance gains. Examining the learned source task weights in the 1k setting, we find that weights for CP, DP and NER have values 0.41, 0.41 and 0.18 respectively which shows that SRL benefits greatly from syntactic tasks like CP and DP. This is in agreement with SRL state-of-the-art models BID19 and BID8 which rely on syntactic features. When we replace GloVe with ELMo representations, we see that the baseline model improves by over ∼ 13%, showing that ELMo representations are indeed very strong. But adding supervised embeddings in the 1k setting further improves upon the ELMo baseline by over ∼ 5%. A similar improvement of ∼ 5% can be seen in the 5k setting as well. Our model shows comparable performance as the baseline when we use the entire dataset. These suggest that the proposed supervised contextual embeddings further bring about improvements over already strong language model features in a low-resource setting. This reinforces the learning that when sufficient data is available, supervised signals do not provide information that the model cannot learn by itself from the data alone. Cross-domain NER Supervised embeddings provide an impressive 4% improvement over the GloVe baseline with both 1,000 and 5,000 samples. Even when we replace GloVe with ELMo, we see an improvement of 3%, indicating that the benefits of using knowledge from other domains is orthogonal to what ELMo can offer. However, the gains vanish when the full dataset is used, suggesting that knowledge from other domains is particularly useful in the very low-resource setting. However, if sufficient data is available, the model has enough resources to build upon generic word embeddings. It is also interesting to note that for this dataset, GloVe based models outperform their ELMo counterparts. This is probably due to the mismatch in the data used to train ELMo (formal language from the 1 billion word corpus) as opposed to the NER dataset which consists of informal language used in web blogs. Cross-lingual NER We observe substantial gains by exploiting information present in other languages. For both German and Spanish the performance gains are highest when number of samples is 1,000, thus validating the suitability of the proposed method for transfer to very low-resource settings. Even when full dataset is used, we see gains over 1% for both languages. We propose supervised contextual embeddings, an easy way to incorporate supervised knowledge from multiple pre-existing models. We perform experiments in the cross-task, cross-domain and cross-lingual setups and find that the proposed embeddings are particularly useful in the lowresource setting. Our work points to the potential of such embeddings in various downstream tasks in different transfer learning settings. Future work includes incorporating more tasks, domains and languages, and understanding the relationships among them. These explorations would build towards our larger vision of building a more complete taxonomy of transfer learning dependencies among NLP tasks, domains and languages.' == 'extract contextual embeddings from off-the-shelf supervised model. Helps downstream NLP models in low-resource settings' != 'We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.'

Question: Deep networks have achieved impressive across a variety of important tasks. However, a known weakness is a failure to perform well when evaluated on data which differ from the training distribution, even if these differences are very small, as is the case with adversarial examples. We propose \emph{Fortified Networks}, a simple transformation of existing networks, which “fortifies” the hidden layers in a deep network by identifying when the hidden states are off of the data manifold, and maps these hidden states back to parts of the data manifold where the network performs well. Our principal contribution is to show that fortifying these hidden states improves the robustness of deep networks and our experiments (i) demonstrate improved robustness to standard adversarial attacks in both black-box and white-box threat models; (ii) suggest that our improvements are not primarily due to the problem of deceptively good due to degraded quality in the gradient signal (the gradient masking problem) and (iii) show the advantage of doing this fortification in the hidden layers instead of the input space. We demonstrate improvements in adversarial robustness on three datasets (MNIST, Fashion MNIST, CIFAR10), across several attack parameters, both white-box and black-box settings, and the most widely studied attacks (FGSM, PGD, Carlini-Wagner). We show that these improvements are achieved across a wide variety of hyperparameters. The success of deep neural networks across a variety of tasks has also driven applications in domains where reliability and security are critical, including self-driving cars BID6, health care, face recognition BID25, and the detection of malware BID17. Security concerns arise when an agent using such a system could benefit from the system performing poorly. Reliability concerns come about when the distribution of input data seen during training can differ from the distribution on which the model is evaluated. Adversarial examples BID11 from attacks on neural network models, applying small perturbations to the inputs that change the predicted class. Such perturbations can be small enough to be unnoticeable to the naked eye. It has been shown that gradient-based methods allow one to find modifications of the input that often change the predicted class BID26 BID11. More recent work demonstrated that it is possible to create modifications such that even when captured through a camera, they change the predicted class with high probability BID7.Some of the most prominent classes of defenses against adversarial examples include feature squeezing BID29, adapted encoding of the input (Jacob BID14, and distillation-related approaches BID20 . Existing defenses provide some robustness but most are not easy to deploy. In addition, many have been shown to be providing the illusion of defense by lowering the quality of the gradient signal, without actually providing improved robustness BID1 . Still others require training a generative model directly in the visible space, which is still difficult today even on relatively simple datasets. Our work differs from the approaches using generative models in the input space in that we instead employ this robustification on the distribution of the learned hidden representations, which makes the The plot on the right shows direct experimental evidence for this hypothesis: we added fortified layers with different capacities to MLPs trained on MNIST, and display the value of the total reconstruction errors for adversarial examples divided by the total reconstruction errors for clean examples. A high value indicates success at detecting adversarial examples. Our support the central motivation for fortified networks: that off-manifold points can much more easily be detected in the hidden space (as seen by the relatively constant ratio for the autoencoder in hidden space) and are much harder to detect in the input space (as seen by this ratio rapidly falling to zero as the input-space autoencoder's capacity is reduced).identification of off-manifold examples easier. We do this by training denoising autoencoders on top of the hidden layers of the original network. We call this method Fortified Networks. We demonstrate that Fortified Networks (i) can be generically added into an existing network; (ii) robustify the network against adversarial attacks and (iii) provide a reliable signal of the existence of input data that do not lie on the manifold on which it the network trained. In the sections that follow, we discuss the intuition behind the fortification of hidden layers and lay out some of the method's salient properties. Furthermore, we evaluate our proposed approach on MNIST, Fashion-MNIST, CIFAR10 datasets against whitebox and blackbox attacks. The Empirical Risk Minimization Framework Let us consider a standard classification task with an underlying data distribution D over pairs of examples x ∈ R d and corresponding labels y ∈ [k]. We also assume that we are given a suitable loss function L(θ, x, y), for instance the cross-entropy loss. As usual, θ ∈ R p is the set of model parameters. Our goal then is to find model parameters θ that minimize the risk E (x,y)∼D [L(x, y, θ) ]. This expectation cannot be computed, therefore a common approach is to to minimize the empirical risk 1/N D L(x, y, θ) taking into account only the examples in a given dataset D.Adversarial Attacks and Robustness While the empirical risk minimization framework has been very successful and often leads to excellent generalization, it has the significant limitation that it doesn't guarantee robustness, and more specifically performance on examples off the data manifold. BID19 proposed an optimization view of adversarial robustness, in which the adversarial robustness of a model is defined as a min-max problem, DISPLAYFORM0 where S denotes the set of all points within a sphere of radius ε, which is task-specific. Larger ε values correspond to stronger attacks but which may be more visually apparent. Denoising Autoencoders Denoising autoencoders (DAEs) are neural networks which take a noisy version of an input (for example, an image) and are trained to predict the noiseless version of that input. This approach has been widely used for feature learning and generative modeling in deep learning BID4. More formally, denoising autoencoders are trained to minimize a reconstruction error or negative log-likelihood of generating the clean input. For example, with DISPLAYFORM1 Figure 2: Diagram illustrating a one-layer fortified network. A network is evaluated with a data sample x and its corresponding adversarial example x. Hidden units h k and h k are corrupted with noise, encoded with the encoder enc., and decoded with the decoder dec. The autoencoder (denoted by the red color) is trained to reconstruct the hidden unit h k that corresponds to the clean input. Dotted lines are two reconstruction costs: for a benign (L rec) and adversarial examples (L adv).Gaussian log-likelihood of the clean input given the corrupted input, r θ the learned denoising function, C a corruption function with Gaussian noise of variance σ 2, the reconstruction loss is BID0 demonstrated that with this loss function, an optimally trained denoising autoencoder's reconstruction vector is proportional to the gradient of the log-density: DISPLAYFORM2 DISPLAYFORM3 This theory establishes that the reconstruction vectors from a well-trained denoising autoencoder form a vector field which points in the direction of the data manifold. However, BID0 showed that this may not hold for points which are distant from the manifold, as these points are rarely sampled during training. In practice, denoising autoencoders are not just trained with tiny noise but also with large noises, which blurs the data distribution as seen by the learner but makes the network learn a useful vector field even far from the data. We propose the use of DAEs inserted at crucial points between layers of the original neural network in order to denoise the transformed data points which may lie away from the original data manifold. Intuitively, the method aims to regularize the hidden representations by keeping the activations on the surface of the corresponding projected data manifold through the application of a DAE trained on the hidden representations (on the original clean data). We argue that applying the DAEs on the hidden layers-as opposed to the raw input signal-facilitates learning, while providing a stronger protection from adversarial attacks. As illustrated in FIG0, we hypothesize that more abstract representations associated with deeper networks are easier to denoise because the transformed data manifolds are flatter. The flattening of data manifolds in the deeper layers of a neural network was first noted experimentally by BID5. We provide experimental support for these claims in Section 4.Layer fortification R Our method works by substituting a hidden layer h k with a denoised version. We feed the signal h k through the encoder network, E k, and decoder network, D k, of a DAE for layer k, which yields the denoised version, h DISPLAYFORM0 where n k is white Gaussian noise of variance σ 2 and appropriate shape. We call the ing layer, a fortified layer and the ing network the fortified network corresponding to the original network.• Reconstruction loss. For a mini-batch of N clean examples, DISPLAYFORM1 is fed into a DAE loss, similar to equation 2: DISPLAYFORM2 • Adversarial loss. We use some adversarial training method to produce the perturbed version of the mini-batch, DISPLAYFORM3, where x (i) is a small perturbation of x (i) which is designed to make the network produce the wrong answer. The corresponding hidden layer h DISPLAYFORM4 (using the perturbed rather than original input) is fed into a similar DAE loss: DISPLAYFORM5 where we note that the target reconstruction for denoising is the clean version of the hidden layer, without noise and without adversarial perturbation. For training purposes, we treat the DAEs as part of the fortified network, backpropagate through and train all weights jointly. Aside from the original classification loss, L c, we also include the classification loss from the adversarial objective, L c (y), and we introduce a dual objective for the DAEs. To build a fortified network, we can apply this fortification process to some or all the layers. The final objective used for training the fortified network includes the classification loss and all reconstruction and adversarial losses: DISPLAYFORM6 where λ rec > 0 and λ adv > 0 tune the strength of the DAE terms. This kind of training process allows for the production of hidden representations robust to small perturbations, and in particular, to adversarial attacks. Off-manifold signaling The reconstruction losses act as a reliable signal for detecting off-manifold examples (cf. Section 4). This is a particularly useful property in practice: not only can we provide more robust classification , we can also sense and suggest to the analyst or system when the original example is either adversarial or from a significantly different distribution. Motivation for when and where to use fortified layers We have discussed advantages to placing fortified layers in the hidden states instead of the input space (with further discussion in section 6,) but the question of where exactly fortified layers need to be placed remains unanswered. Is it just the final hidden layer? Is it every hidden layer? We outline two important considerations regarding this issue. First, in the higher-level hidden layers, it is much easier for the network to identify points which are off of the manifold or close to the margin. This is directly experimentally demonstrated in fig. 1. Secondly, the higher level hidden layers may already look like points that are not adversarial due to the effect of the adversarial perturbations in the earlier layers. While we are not aware of any formal study of this phenomenon, it is clearly possible (imagine for example a fortified layer on the output from the softmax, which could only identify unnatural combinations of class probabilities.) Given these opposing objectives, we argue for the inclusion of multiple fortified layers across the network. We evaluated the performance of our model as a defense against adversarial attacks. We focused on two of the most popular and well-studied attacks. Firstly, we consider the Fast Gradient Sign Method (FGSM, BID11 which is popular as it only requires a single step and can still be effective against many networks. Secondly, we consider the projected gradient descent attack BID16 which is slower than FGSM as it requires many iterations, but has been shown to be a much stronger attack BID19 .Additionally, we consider both white-box attacks (where the attackers knows the model) and blackbox attacks (where they don't, but they have access to the training set.)Fast Gradient Sign Method. The Fast Gradient Sign Method (FGSM) BID11 is a simple one-step attack that produces ∞ -bounded adversaries via the following gradient based perturbation. DISPLAYFORM0 Projected Gradient Descent. The projected gradient descent attack BID19, sometimes referred to as FGSM k, is a multi-step extension of the FGSM attack characterized as follows: DISPLAYFORM1 initialized with x 0 as the clean input x and with the corrupted input x as the last step in the sequence. Π refers to the projection operator, which in this context means projecting the adversarial example back onto the region within an ε radius of the original data point, after each step in the adversarial attack. Finally we considered the Carlini-Wagner L2 attack BID8 which consists of joint optimization of loss maximization and minimizing the distance of the adversarial example to the original example. A significant challenge with evaluating defenses against adversarial attacks is that many attacks rely upon a network's gradient. Methods which reduce the quality of this gradient, either by making it flatter or noisier can lead to methods which lower the effectiveness of gradient-based attacks, but which are not actually robust to adversarial examples BID2 BID23. This process, which has been referred to as gradient masking or gradient obfuscation, must be analyzed when studying the strength of an adversarial defense. One method for studying the extent to which an adversarial defense gives deceptively good as a of gradient masking relies on the observation that black-box attacks are a strict subset of white-box attacks, so white-box attacks should always be at least as strong as black-box attacks. If a method reports much better defense against white-box attacks, it suggests that the selected white-box attack is underpowered as a of gradient masking. Another test for gradient masking is to run an iterative search, such as projected gradient descent (PGD) with an unlimited range for a large number of iterations. If such an attack is not completely successful, it indicates that the model's gradients are not an effective method for searching for adversarial images, and that gradient masking is occurring. Still another test is to confirm that iterative attacks with small step sizes always outperform single-step attacks with larger step sizes (such as FGSM). If this is not the case, it may suggest that the iterative attack becomes stuck in regions where optimization using gradients is poor due to gradient masking. Additionally, BID1 discussed the Backward Pass Differentiable Approximation (BPDA) attack to cover cases where a defense employs a transformation which is clearly nondifferentiable or reduces the quality of the gradients. Because we pass gradients through the fortified layers in the normal training of our network, it is unlikely that the quality of these gradients is significantly deteriorated, and there isn't a reason to expect that they would be because the fortified layers are relatively shallow and use normal activation functions (i.e. no non-differentiable functions.). Additionally we ran additional experiments using the identity function version of BPDA on Fashion MNIST FGSM (ε = 0.3) with standard deviation across five trials. Our adversarial training baseline achieved an accuracy of 84.64±0.48 on this task, fortified networks with the normal attack achieved an accuracy of 89.50±0.45, and fortified networks with the BPDA version of the attack (treating the autoencoder as an identity) achieved an accuracy of 89.88±0.30, which corresponds to a weaker attack. For details about the specifics of our model architectures and hyperparameters we refer readers to the Appendix. With all experiments, we use the same attacks (with identical parameters) at training and test time to generate adversarial examples. An important point to note here is that all of the autoencoders in our fortified layers used a single hidden layer with tied weights. We also performed an analytical experiment on an RNN language model to study if fortified networks can detect when it's being given outputs from its own model (sampling mode) when trained using ground truth input sequences. To this end we train a language model on the standard Text8 dataset, which is derived from Wikipedia articles. We trained a single-layer LSTM with 1000 units at the character-level, and included fortified layers between the hidden states and the output on each time step. With 50 sampling steps, the fortified layers had a reconstruction error on average 103% of the teacher forcing reconstruction error. With 180 sampling steps, this valued increased to 112%. With 300 sampling steps this increased even further to 134%. This is clear evidence that the outputs move off of the manifold with more sampling steps, and that this is effectively measured by fortified networks. We ran with many hyperparameters for the fortified layers to demonstrate the generality of the improvement given by fortified networks. We ran FGSM on Fashion-MNIST (ε=0.3) while varying the amount of noise injected and the weighting on the reconstruction loss. We also varied the amount of noise and choice of losses in the ablation experiment in Table 7. We achieved consistent improvement across a variety of settings but found the largest improvement when using both the adversarial and clean reconstruction losses and a small amount of noise. Thus, we see that a consistent improvement is achieved when the weighting on the reconstruction part of the loss and the amount of noise injected are varied over several orders of magnitude. Using Generative Models as a Defense. The observation that adversarial examples often consist of points off of the data manifold and that deep networks may not generalize well to these points motivated BID12 BID24 BID18 to consider the use of the generative models as a defense against adversarial attacks. Ilyas et al. FORMULA0; BID10 also showed the existence of adversarial examples which lie on the data manifold, and showed that training against adversarial examples forced to lie on the manifold is an effective defense. Our method shares a closely related motivation to these prior works, with a key difference being that we propose to consider the manifold in the space of learned representations, Table 2: CIFAR-10 PGD Results with (non-resnet) CNNs. In these experiment we used a fortified block (single convolutional autoencoder) following each convolutional layer. Both experiments were run for 200 epochs and with all hyperparameters and architecture kept the same with the exception of the fortified layer being added. We considered different types of baselines:'Baseline -no new layers' means we simply removed the fortified block.' Baseline -extra layers' means that we added extra layers to match the capacity of the fortified layers, but only gave half of these extra layers activations as the fortified block has two layers but only one activation.' Baseline -extra activations' means that we added an activation following each layer, giving more activations in total than the Fortified Network. Attack instead of considering the manifold directly in the visible space. One motivation for this is that the learned representations have a simpler statistical structure, which makes the task of modeling this manifold and detecting unnatural points much simpler. Learning the distribution directly in the visible space is still very difficult (even state of the art models fall short of real data on metrics like Inception Score) and requires a high capacity model. Additionally, working in the space of learned representations allows for the use of a relatively simple generative model, in our case a small denoising autoencoder. proposed to work around these challenges from working in the visible space by using the Deep Image Prior instead of an actual generative model. While this has the advantage of Table 4: Accuracies against white-box attacks on Fashion MNIST. For PGD we used ε = 0.1 and for FGSM we used ε = 0.1 and ε = 0.3 1. Compared with DefenseGAN BID24. Table 5: Left: Accuracies against blackbox MNIST attacks with adversarial training (FGSM). DISPLAYFORM0 Reporting 50/50 compared to previous works (, PS). The test error on clean examples is in parenthesis. Right:We ran a fortified network on Fashion-MNIST using adversarial training with PGD for a variety of ε values, each for 5 epochs. The motivation behind this experiment, suggested by BID1 is confirming if unbounded (ε = 1) adversarial attacks are able to succeed. A defense which succeeds primarily by masking or obfuscating the gradients would fail to bring the accuracy to zero even with an unbounded attack. As can be seen, unbounded attacks against Fortified Networks succeed when given a sufficiently large ε, which is evidence against gradient masking. being a model that doesn't require a special training procedure (as deep image prior is a separate optimization process for each example) it may be limited in the types of adversarial attacks that it's resistant to, and it would provide no defense against adversarial attacks which are in the range of a convolutional network, which have been shown to exist BID28.Another key difference between our work and BID24 is that both DefenseGAN and the Invert-and-Classify approach use an iterative search procedure at inference time to map observed data points onto nearby points on the range of the generator. On the other hand, our approach uses small denoising autoencoders that are used in the same way (i.e. a simple forward application) during both training and testing. The use of such an iterative procedure presents challenges for evaluation, as it is possible for gradients to vanish while doing backpropagation through such a procedure, which may lead to an overestimate in the strength of the defense due to the gradient masking problem BID22 BID1. One indicator of the gradient masking problem is black-box attacks outperforming white-box attacks, which is an indicator of under-powered attacks as black-box attacks are a strict subset of white-box attacks. This indicator of gradient obfuscation was present in the work of BID24 where black-box attacks were generally stronger against their defense, but with our method we observe very similar defense quality against black-box and white-box attacks. BID12 BID18 both considered using an autoencoder as a pre-processing step in the input space. Interestingly BID18 ) used a loss function defined in the space of the hidden states, but still used autoencoders directly in the input space. Adversarial Hidden State Matching. BID9 demonstrate that adversarially matching the hidden layer activations of regular and adversarial examples improves robustness. This work shared the same motivation of using the hidden states to improve robustness, but differed in that they used an adversarial objective and worked in the original hidden states instead of using a generative model (in our case, the DAE in the fortified layers). We present direct experimental comparisons with their work in section 5. BID15 proposed a method which involves matching the logit (pre-softmax outputs) values for the original samples with the logit values ing from adversarial examples. Denoising Feature Matching. BID27 proposed to train a denoising autoencoder in the hidden states of the discriminator in a generative adversarial network. The generator's parameters are then trained to make the reconstruction error of this autoencoder small. This has the effect of encouraging the generator to produce points which are easy for the model to reconstruct, which will include true data points. Both this and Fortified Networks use a learned denoising autoencoder in the hidden states of a network. A major difference is that the denoising feature matching work focused on generative adversarial networks and tried to minimize reconstruction error through a learned generator network, whereas our approach targets the adversarial examples problem. Additionally, our objective encourages the output of the DAE to denoise adversarial examples so as to point back to the hidden state of the original example, which is different from the objective in the denoising feature matching work, which encouraged reconstruction error to be low on states from samples from the generator network. Adversarial Spheres. BID10 studied the existence of adversarial examples in the task of classifying between two hollow concentric shells. Intriguingly, they prove and construct adversarial examples which lie on the data manifold (although also looked for such examples experimentally using GANs.) The existence of such on-manifold adversarial examples demonstrates that a simplified version of our model trained with only L rec could not protect against all adversarial examples. However, training with L adv encourages the fortified layers to map back from points which are not only off of the manifold, but also to map back from points which are hard to classify, allowing Fortified Networks to also potentially help with on-manifold adversarial examples as well. Protecting against adversarial examples could be of paramount importance in mission-critical applications. We have presented Fortified Networks, a simple method for the robustification of existing deep neural networks. Our method is practical, as fortifying an existing network entails introducing DAEs between the hidden layers of the network, which can be automated. Furthermore, the DAE reconstruction error at test time is a reliable signal of distribution shift, which can in examples unlike those encountered during training. High error can signify either adversarial attacks or significant domain shift; both are important cases for the analyst or system to be aware of. Moreover, fortified networks are efficient: since not every layer needs to be fortified to achieve improvements, fortified networks are an efficient way to improve robustness to adversarial examples. For example, we have shown improvements on ResNets where only two fortified layers are added, and thus the change to the computational cost is very slight. Finally, fortified networks are effective, as they improve on adversarial defense on three datasets (MNIST, Fashion MNIST, and CIFAR10), across a variety of attack parameters (including the most widely used ε values), across three widely studied attacks (FGSM, PGD, Carlini-Wagner L2), and in both the black-box and white-box settings. A EXPERIMENTAL SETUP All attacks used in this work were carried out using the Cleverhans BID21 ) library. A.1 WHITE-BOX ATTACKS Our convolutional models (Conv, in the tables) have 2 strided convolutional layers with 64 and 128 filters followed by an unstrided conv layer with 128 filters. We use ReLU activations between layers then followed by a single fully connected layer. The convolutional and fully-connected DAEs have a single bottleneck layer with leaky ReLU activations with some ablations presented in the table below. With white-box PGD attacks, we used only convolutional DAEs at the first and last conv layers with Gaussian noise of σ = 0.01 whereas with FGSM attacks we used a DAE only at the last fully connected layer. The weight on the reconstruction error λ rec and adversarial cost λ adv were set to 0.01 in all white-box attack experiments. We used the Adam optimizer with a learning rate of 0.001 to train all models. The table below lists a few ablations with different activation functions in the autoencoder Our black-box are based on a fully-connected substitute model (input-200-200-output), which was subsequently used to attack a fortified convolutional network. The CNN was trained for 50 epochs using adversarial training, and the predictions of the trained CNN were used to train the substitute model. 6 iterations of Jacobian data augmentation were run during training of the substitute, with λ = 0.1. The test set data holdout for the adversary was fixed to 150 examples. The learning rate was set to 0.003 and the Adam optimizer was used to train both models. TAB0: More attack steps to uncover gradient masking effects. Different epsilon values at attack time. We applied attacks of different ε values to a network trained with ε = 0.03. Results shown in tab. 9.More attack iterations. We ran attacks for more steps at test time, on a network trained on attacks of 7 steps. Results shown in tab. 10.

Represent the short summary for finding the corresponding scientific article (long).

Better adversarial training by learning to map back to the data manifold with autoencoders in the hidden states.

To understand the information stored in the latent space, we train a GAN-style decoder constrained to produce images that the VAE encoder will map to the same region of latent space.


Question: Represent the scientific article to find a short TLDR

Question: Neural networks could misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset. In this work, we study the binary classification of linearly separable datasets and show that linear classifiers could also have decision boundaries that lie close to their training dataset if cross-entropy loss is used for training. In particular, we show that if the features of the training dataset lie in a low-dimensional affine subspace and the cross-entropy loss is minimized by using a gradient method, the margin between the training points and the decision boundary could be much smaller than the optimal value. This is contrary to the of recent related works such as , and we identify the reason for this contradiction. In order to improve the margin, we introduce differential training, which is a training paradigm that uses a loss function defined on pairs of points from each class. We show that the decision boundary of a linear classifier trained with differential training indeed achieves the maximum margin. The reveal the use of cross-entropy loss as one of the hidden culprits of adversarial examples and introduces a new direction to make neural networks robust against them. Training neural networks is challenging and involves making several design choices. Among these are the architecture of the network, the training loss function, the optimization algorithm used for training, and their hyperparameters, such as the learning rate and the batch size. Most of these design choices influence the solution obtained by the training procedure and have been studied in detail BID9 BID4 BID5; BID17 BID19. Nevertheless, one choice has been mostly taken for granted when the network is trained for a classification task: the training loss function. Cross-entropy loss function is almost the sole choice for classification tasks in practice. Its prevalent use is backed theoretically by its association with the minimization of the Kullback-Leibler divergence between the empirical distribution of a dataset and the confidence of the classifier for that dataset. Given the particular success of neural networks for classification tasks BID11 BID18 BID5, there seems to be little motivation to search for alternatives for this loss function, and most of the software developed for neural networks incorporates an efficient implementation for it, thereby facilitating its use. Recently there has been a line of work analyzing the dynamics of training a linear classifier with the cross-entropy loss function BID15 b; BID7. They specified the decision boundary that the gradient descent algorithm yields on linearly separable datasets and claimed that this solution achieves the maximum margin.1 However, these claims were observed not to hold in the simple experiments we ran. For example, FIG6 displays a case where the cross-entropy minimization for a linear classifier leads to a decision boundary which attains an extremely poor margin and is nearly orthogonal to the solution given by the hard-margin support vector machine (SVM).We set out to understand this discrepancy between the claims of the previous works and our observations on the simple experiments. We can summarize our contributions as follows. Cross-entropy min. decision boundary (poor margin)Figure 1: Orange and blue points represent the data from two different classes in R 2. Cross-entropy minimization for a linear classifier on the given training points leads to the decision boundary shown with the solid line, which attains a very poor margin and is almost orthogonal to the solution given by the SVM.1. We analyze the minimization of the cross-entropy loss for a linear classifier by using only two training points, i.e., only one point from each of the two classes, and we show that the dynamics of the gradient descent algorithm could yield a poor decision boundary, which could be almost orthogonal to the boundary with the maximum margin.2. We identify the source of discrepancy between our observations and the claims of the recent works as the misleading abbreviation of notation in the previous works. We clarify why the solution obtained with cross-entropy minimization is different from the SVM solution.3. We show that for linearly separable datasets, if the features of the training points lie in an affine subspace, and if the cross-entropy loss is minimized by a gradient method with no regularization to train a linear classifier, the margin between the decision boundary of the classifier and the training points could be much smaller than the optimal value. We verify that when a neural network is trained with the cross-entropy loss to classify two classes from the CIFAR-10 dataset, the output of the penultimate layer of the network indeed produces points that lie on an affine subspace.4. We show that if there is no explicit and effective regularization, the weights of the last layer of a neural network could grow to infinity during training with a gradient method. Even though this has been observed in recent works as well, we are the first to point out that this divergence drives the confidence of the neural network to 100% at almost every point in the input space if the network is trained for long. In other words, the confidence depends heavily on the training duration, and its exact value might be of little significance as long as it is above 50%.5. We introduce differential training, which is a training paradigm that uses a loss function defined on pairs of points from each class -instead of only one point from any class. We show that the decision boundary of a linear classifier trained with differential training indeed produces the SVM solution with the maximum hard margin. We start with a simple binary classification problem. Given two points x ∈ R d and −y ∈ R d from two different classes, we can find a linear classifier by minimizing the cross-entropy loss function log(e −w x + 1) + log(e −w ỹ + 1), DISPLAYFORM0. Unless the two points x and −y are equal, the function does not attain its minimum at a finite value ofw. Consequently, if the gradient descent algorithm is used to minimize, the iterate at time k,w[k], diverges as k increases. The following theorem characterizes the growth rate ofw[k] and its direction in the limit by using a continuous-time approximation to the gradient descent algorithm. Theorem 1. Given two points x ∈ R d and −y ∈ R d, letx and −ỹ denote [x 1] and [−y 1], respectively. Without loss of generality, assume x ≤ y. If the two points are in different classes and we minimize the cross-entropy loss DISPLAYFORM1 where σ x = x 2, σ xy =x ỹ and σ y = ỹ 2.Note that first d coordinates of represent the normal vector of the decision boundary obtained by minimizing the cross-entropy loss. This vector is different from x + y, which is the direction of the maximum-margin solution given by the SVM. In fact, the direction in could be almost orthogonal to the SVM solution in certain cases, which implies that the margin between the points and the decision boundary could be much smaller than the optimal value. Corollary 1 describes a subset of these cases. Corollary 1. Given two points x and −y in R d, let ψ denote the angle between the solution given by and the solution given by the SVM, i.e., (x + y). If x y = 1, then DISPLAYFORM2 where σ x = x 2 +1 and σ y = y 2 +1. Consequently, as x / y approaches 0 while maintaining the condition x y = 1, the angle ψ converges to π/2. Remark 1. Corollary 1 shows that if x and −y have disparate norms, the minimization of the crossentropy loss with gradient descent algorithm could lead to a direction which is almost orthogonal to the maximum-margin solution. It may seem like this problem could be avoided with preprocessing the data so as to normalize the data points. However, this approach will not be effective for neural networks: if we consider an L-layer neural network, w φ L−1 (x), and regard the first L − 1 layers, φ L−1 (·), as a feature mapping, preprocessing a dataset {x i} i∈I will not produce a normalized set of features {φ L−1 (x i)} i∈I. Note that we could not normalize {φ L−1 (x i)} i∈I directly either, since the mapping φ L−1 (·) evolves during training. Remark 2. Theorem 1 shows that the norm of w keeps growing unboundedly as the training continues. The same behavior will be observed for larger datasets in the next sections as well. Since the "confidence" of the classifier for its prediction at a point x is given by DISPLAYFORM3 this unbounded growth of w drives the confidence of the classifier to 100% at every point in the input space, except at the points on the decision boundary, if the algorithm is run for long. Given the lack of effective regularization for neural networks, a similar unbounded growth is expected to be observed in neural network training as well, which is mentioned in BID1. As a , the confidence of a neural network might be highly correlated with the training duration, and whether a neural network gives 99% or 51% confidence for a prediction might be of little importance as long as it is above 50%. In other words, regarding this confidence value as a measure of similarity between an input and the training dataset from the most-likely class should be reconsidered. In this section, we examine the binary classification of a linearly separable dataset by minimizing the cross-entropy loss function. Recently, this problem has also been studied in BID16 a; BID7. We restate an edited version of the main theorem of, followed by the reason of the edition. Theorem 2 (Adapted from Theorem 3 of). Given two sets of points {x i} i∈I and {−y j} j∈J that are linearly separable in R d, letx i and −ỹ j denote [x i 1] and [−y j 1], respectively, for all i ∈ I, j ∈ J. Then the iterate of the gradient descent algorithm,w(t), on the cross-entropy loss function DISPLAYFORM0 with a sufficiently small step size will converge in direction: DISPLAYFORM1 where w is the solution to DISPLAYFORM2 The solution given in Theorem 2 was referred in, and consequently in the other works, as the maximum-margin solution. However, due to the misleading absence of the bias term in the notation, this is incorrect. Given the linearly separable sets of points {x i} i∈I and {−y j} j∈J, the maximum-margin solution given by the SVM solves DISPLAYFORM3 On the other hand, the solution given by Theorem 2 corresponds to DISPLAYFORM4 Even though the sets of constraints for both problems are identical, their objective functions are different, and consequently, the solutions are different. As a , the decision boundary obtained by crossentropy minimization does not necessarily attain the maximum hard margin. In fact, as the following theorem shows, its margin could be arbitrarily worse than the maximum margin. Theorem 3. Assume that the points {x i} i∈I and {−y j} j∈J are linearly separable and lie in an affine subspace; that is, there exist a set of orthonormal vectors {r k} k∈K and a set of scalars DISPLAYFORM5 Let w, · + B = 0 denote the decision boundary obtained by minimizing the cross-entropy loss, i.e. the pair (w, B) solves DISPLAYFORM6 Then the minimization of the cross-entropy loss yields a margin smaller than or equal to DISPLAYFORM7 where γ denotes the optimal hard margin given by the SVM solution. Remark 3. Theorem 3 shows that if the training points lie in an affine subspace, the margin obtained by the cross-entropy minimization will be smaller than the optimal margin value. As the dimension of this affine subspace decreases, the cardinality of the set K increases and the term k∈K ∆ 2 k could become much larger than 1/γ 2. Therefore, as the dimension of the subspace containing the training points gets smaller compared to the dimension of the input space, cross-entropy minimization with a gradient method becomes more likely to yield a poor margin. Note that this argument also holds for classifiers of the form w φ(x) with the fixed feature mapping φ(·).The next theorem relaxes the condition of Theorem 3 and allows the training points to be near an affine subspace instead of being exactly on it. Note that the ability to compare the margin obtained by cross-entropy minimization with the optimal value is lost. Nevertheless, it highlights the fact that same set of points could be assigned a different margin by cross-entropy minimization if all of them are shifted away from the origin by the same amount in the same direction. Theorem 4. Assume that the points {x i} i∈I and {−y j} j∈J in R d are linearly separable and there exist a set of orthonormal vectors {r k} k∈K and a set of scalars {∆ k} k∈K such that DISPLAYFORM8 Let w, · + B = 0 denote the decision boundary obtained by minimizing the cross-entropy loss, i.e. the pair (w, B) solves DISPLAYFORM9 Then the minimization of the cross-entropy loss yields a margin smaller than or equal to DISPLAYFORM10 Remark 4. Both Theorem 3 and Theorem 4 consider linearly separable datasets. If the dataset is not linearly separable, BID7 predicts that the normal vector of the decision boundary, w, will have two components, one of which converges to a finite vector and the other diverges. The diverging component still has the potential to drive the decision boundary to a direction with a poor margin. In fact, the margin is expected to be small especially if the points intruding into the opposite class lie in the same subspace as the optimal normal vector for the decision boundary. In this work, we focus on the case of separable datasets as this case provides critical insight into the issues of state-of-the-art neural networks, given they can easily attain zero training error even on randomly generated datasets, which indicates the linear separability of the features obtained at their penultimate layers . In previous sections, we saw that the cross-entropy minimization could lead to poor margins, and the main reason for this was the appearance of the bias term in the objective function of (P2). In order to remove the effect of the bias term, consider the SVM problem (P1) and note that this problem could be equivalently written as minimize w w 2 2 subject to w, x i + y j ≥ 2 ∀i ∈ I, ∀j ∈ Jif we only care about the weight parameter w. This gives the hint that if we use the set of differences {x i + y j : i ∈ I, j ∈ J} instead of the individual sets {x i} i∈I and {−y j} j∈J, the bias term could be excluded from the problem. This was also noted in BID8 BID6 previously. Indeed, this approach allows obtaining the SVM solution with a loss function similar to the cross-entropy loss, as the following theorem shows. Theorem 5. Given two sets of points {x i} i∈I and {−y j} j∈J that are linearly separable in R d, if we solve min DISPLAYFORM0 by using the gradient descent algorithm with a sufficiently small learning rate, the direction of w converges to the direction of maximum-margin solution, i.e. DISPLAYFORM1 where w SVM is the solution of (P3).Proof. Apply Theorem 2 by replacing the sets {x i} i∈I and {−y j} j∈J with {x i + y j} i∈I,j∈J and the empty set, respectively. Then the minimization of the loss function FORMULA16 Since w SVM is the solution of (P3), we obtain w = 1 2 w SVM, and the claim of the theorem holds. Remark 5. Theorem 5 is stated for the gradient descent algorithm, but the identical statement could be made for the stochastic gradient method as well by invoking the main theorem of BID20.Minimization of the cost function yields the weight parameterŵ of the decision boundary. The bias parameter, b, could be chosen by plotting the histogram of the inner products {ŵ, x i} i∈I and {ŵ, −y j} j∈J and fixing a value forb such that DISPLAYFORM2 DISPLAYFORM3 The largest hard margin is achieved bŷ DISPLAYFORM4 However, by choosing a larger or smaller value forb, it is possible to make a tradeoff between the Type-I and Type-II errors. The cost function includes a loss defined on every pair of data points from the two classes. This cost function can be considered as the cross-entropy loss on a new dataset which contains |I| × |J| points. There are two aspects of this fact:1. When standard loss functions are used for classification tasks, we need to oversample or undersample either of the classes if the training dataset contains different number of points from different classes. This problem does not arise when we use the cost function. 2. Number of pairs in the new dataset, |I| × |J|, will usually be much larger than the original dataset, which contains |I| + |J| points. Therefore, the minimization of might appear more expensive than the minimization of the standard cross-entropy loss computationally. However, if the points in different classes are well separated and the stochastic gradient method is used to minimize, the algorithm achieves zero training error after using only a few pairs, which is formalized in Theorem 6. Further computation is needed only to improve the margin of the classifier. In addition, in our experiments to train a neural network to classify two classes from the CIFAR-10 dataset, only a few percent of |I| × |J| points were observed to be sufficient to reach a high accuracy on the training dataset. Theorem 6. Given two sets of points {x i} i∈I and {−y j} j∈J that are linearly separable in R d, assume the cost function is minimized with the stochastic gradient method. Define R x = max{x i − x i : i, i ∈ I}, R y = max{y j − y j : j, j ∈ J} and let γ denote the hard margin that would be obtained with the SVM: DISPLAYFORM5 If 2γ ≥ 5 max(R x, R y), then the stochastic gradient algorithm produces a weight parameter,ŵ, only in one iteration which satisfies the inequalities (7a)-(7b) along with the bias,b, given by. In this section, we present numerical experiments supporting our claims. Differential training. In Figure 2, we show the decision boundaries of two linear classifiers, where one of them is trained by minimizing the cross-entropy loss, and the other through differential training. Unlike the example shown in FIG6, here the data do not exactly lie in an affine subspace. In particular, one of the classes is composed of 10 samples from a normal distribution with mean and variance 25, and the other class is composed of 10 samples from a normal distribution with mean and variance 25. As can be seen from the figure, the cross-entropy minimization yields a margin that is smaller than differential training, even though when the training dataset is not low-dimensional, which is predicted by Theorem 4. Cross-entropy min. boundary Figure 2: Classification boundaries obtained using differential training and cross-entropy minimization. The margin recovered by cross-entropy minimization is worse than differential training even when the training dataset is not low-dimensional. Low-dimensionality. We empirically evaluated if the features obtained at the penultimate layer of a neural network indeed lie in a low-dimensional affine subspace. For this purpose, we trained a convolutional neural network architecture to classify horses and planes from the CIFAR-10 dataset BID10. FIG3 shows the cumulative variance explained for the features that feed into the soft-max layer as a function of the number of principle components used. We observe that the features, which are the outputs of the penultimate layer of the network, lie in a low-dimensional affine subspace, and this holds for a variety of training modalities for the network. This observation is relevant to Remark 3. The dimension of the subspace containing the training points is at most 20, which is much smaller than the dimension of the feature space, 84. Consequently, cross-entropy minimization with a gradient method is expected to yield a poor margin on these features. We compare our with related works and discuss their implications for the following subjects. Adversarial examples. State-of-the-art neural networks have been observed to misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset (; BID3 ; . Our reveal that the combination of gradient methods, cross-entropy loss function and the low-dimensionality of the training dataset (at least in some domain) has a responsibility for this problem. Note that SVM with the radial basis function was shown to be robust against adversarial examples, and this was attributed to the high nonlinearity of the radial basis function in BID3. Given that the SVM uses neither the cross entropy loss function nor the gradient descent algorithm for training, we argue that the robustness of SVM is no surprise -independent of its nonlinearity. Lastly, effectiveness of differential training for neural networks against adversarial examples is our ongoing work. The activations feeding into the soft-max layer could be considered as the features for a linear classifier. Plot shows the cumulative variance explained for these features as a function of the number of principle components used. Almost all the variance in the features is captured by the first 20 principle components out of 84, which shows that the input to the soft-max layer resides predominantly in a low-dimensional subspace. Low-dimensionality of the training dataset. As stated in Remark 3, as the dimension of the affine subspace containing the training dataset gets very small compared to the dimension of the input space, the training algorithm will become more likely to yield a small margin for the classifier. This observation confirms the of BID13, which showed that if the set of training data is projected onto a low-dimensional subspace before feeding into a neural network, the performance of the network against adversarial examples is improved -since projecting the inputs onto a low-dimensional domain corresponds to decreasing the dimension of the input space. Even though this method is effective, it requires the knowledge of the domain in which the training points are low-dimensional. Because this knowledge will not always be available, finding alternative training algorithms and loss functions that are suited for low-dimensional data is still an important direction for future research. Robust optimization. Using robust optimization techniques to train neural networks has been shown to be effective against adversarial examples BID12 BID0. Note that these techniques could be considered as inflating the training points by a presumed amount and training the classifier with these inflated points. Consequently, as long as the cross-entropy loss is involved, the decision boundaries of the neural network will still be in the vicinity of the inflated points. Therefore, even though the classifier is robust against the disturbances of the presumed magnitude, the margin of the classifier could still be much smaller than what it could potentially be. Differential training. We introduced differential training, which allows the feature mapping to remain trainable while ensuring a large margin between different classes of points. Therefore, this method combines the benefits of neural networks with those of support vector machines. Even though moving from 2N training points to N 2 seems prohibitive, it points out that a true classification should in fact be able to differentiate between the pairs that are hardest to differentiate, and this search will necessarily require an N 2 term. Some heuristic methods are likely to be effective, such as considering only a smaller subset of points closer to the boundary and updating this set of points as needed during training. If a neural network is trained with this procedure, the network will be forced to find features that are able to tell apart between the hardest pairs. Nonseparable data. What happens when the training data is not linearly separable is an open direction for future work. However, as stated in Remark 4, this case is not expected to arise for the state-of-the-art networks, since they have been shown to achieve zero training error even on randomly generated datasets , which implies that the features represented by the output of their penultimate layer eventually become linearly separable. A PROOF OF THEOREM 1Theorem 1 could be proved by using Theorem 2, but we provide an independent proof here. Gradient descent algorithm with learning rate δ on the cross-entropy loss yields DISPLAYFORM0 1 + e −w x + δỹ e −w ỹ 1 + e −w ỹ.Ifw = 0, thenw(t) = p(t)x + q(t)ỹ for all t ≥ 0, wherė DISPLAYFORM1 Then we can writeα Lemma 2. If b < 0, then there exists t 0 ∈ (0, ∞) such that DISPLAYFORM2 Proof. Note that DISPLAYFORM3 which implies that DISPLAYFORM4 as long as DISPLAYFORM5 By using Lemma 2, DISPLAYFORM6 Proof. Solving the set of equations DISPLAYFORM7, DISPLAYFORM8 Proof. Note thatż ≥ a/2 andv ≥ c/2; therefore, DISPLAYFORM9 if either side exists. Remember thaṫ DISPLAYFORM10 We can compute f (w) = 2acw + bcw 2 + ab b 2 w 2 + 2abw + a 2. The function f is strictly increasing and convex for w > 0. We have DISPLAYFORM11 Therefore, when b ≥ a, the only fixed point of f over [0, ∞) is the origin, and when a > b, 0 and (a − b)/(c − b) are the only fixed points of f over [0, ∞). Figure 4 shows the curves over whichu = 0 andẇ = 0. Since lim t→∞ u = lim t→∞ w, the only points (u, w) can converge to are the fixed points of f. Remember thaṫ DISPLAYFORM12 so when a > b, the origin is unstable in the sense of Lyapunov, and (u, w) cannot converge to it. Otherwise, is the only fixed point, and it is stable. As a , DISPLAYFORM13 Figure 4: Stationary points of function f. DISPLAYFORM14 Proof. From Lemma 6, DISPLAYFORM15 Consequently, DISPLAYFORM16 which gives the same solution as Lemma 5: DISPLAYFORM17 Proof. We can obtain a lower bound for square of the denominator as DISPLAYFORM18 DISPLAYFORM19 As a , Then, we can write w as DISPLAYFORM20 Remember, by definition, w SVM = arg min w 2 s.t. w, x i + y j ≥ 2 ∀i ∈ I, ∀j ∈ J.Since the vector u also satisfies u, x i + y j = w, x i + y j ≥ 2 for all i ∈ I, j ∈ J, we have u ≥ w SVM = 1 γ. As a , the margin obtained by minimizing the cross-entropy loss is DISPLAYFORM21 If B < 0, we could consider the hyperplane w, · − B = 0 for the points {−x i} i∈I and {y j} j∈J, which would have the identical margin due to symmetry. Therefore, without loss of generality, assume B ≥ 0. As in the proof of Theorem 3, KKT conditions for the optimality of w and B requires w = i∈I µ i x i + j∈J ν j y j, B = i∈I µ i − j∈J ν j where µ i ≥ 0 and ν j ≥ 0 for all i ∈ I, j ∈ J. Note that for each k ∈ K, w, r k = i∈I µ i x i, r k − j∈J ν j −y j, r k DISPLAYFORM0 Since {r k} k∈K is an orthonormal set of vectors, DISPLAYFORM1 The follows from the fact that w −1 is an upper bound on the margin. E PROOF OF THEOREM 6In order to achieve zero training error in one iteration of the stochastic gradient algorithm, it is sufficient to have min i ∈I x i, x i + y j > max j ∈J −y j, x i + y j ∀i ∈ I, ∀j ∈ J, or equivalently, x i + y j, x i + y j > 0 ∀i, i ∈ I, ∀j, j ∈ J.By definition of the margin, there exists a vector w SVM ∈ R d with unit norm which satisfies 2γ = min i∈I,j∈J x i + y j, w SVM.Note that w SVM is orthogonal to the decision boundary given by the SVM. Then we can write every x i + y j as x i + y j = 2γw SVM + δ If we choose γ > 5 2 max(R x, R y), we have 4γ 2 − 2γ(2R x + 2R y) − (R x + R y) 2 > 0, which guarantees and completes the proof.

Represent the short summary for finding the corresponding scientific article (long)

We show that minimizing the cross-entropy loss by using a gradient method could lead to a very poor margin if the features of the dataset lie on a low-dimensional subspace.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.