Question: represent the text to find a very short summary (1-2 sentences max).

Question: Zero-Shot Learning (ZSL) is a classification task where some classes referred as unseen classes have no labeled training images. Instead, we only have side information (or description) about seen and unseen classes, often in the form of semantic or descriptive attributes. Lack of training images from a set of classes restricts the use of standard classification techniques and losses, including the popular cross-entropy loss. The key step in tackling ZSL problem is bridging visual to semantic space via learning a nonlinear embedding. A well established approach is to obtain the semantic representation of the visual information and perform classification in the semantic space. In this paper, we propose a novel architecture of casting ZSL as a fully connected neural-network with cross-entropy loss to embed visual space to semantic space. During training in order to introduce unseen visual information to the network, we utilize soft-labeling based on semantic similarities between seen and unseen classes. To the best of our knowledge, such similarity based soft-labeling is not explored for cross-modal transfer and ZSL. We evaluate the proposed model on five benchmark datasets for zero-shot learning, AwA1, AwA2, aPY, SUN and CUB datasets, and show that, despite the simplicity, our approach achieves the state-of-the-art performance in Generalized-ZSL setting on all of these datasets and outperforms the state-of-the-art for some datasets. Supervised classifiers, specifically Deep Neural Networks, need a large number of labeled samples to perform well. Deep learning frameworks are known to have limitations in fine-grained classification regime and detecting object categories with no labeled data; ). On the contrary, humans can recognize new classes using their previous knowledge. This power is due to the ability of humans to transfer their prior knowledge to recognize new objects . Zero-shot learning aims to achieve this human-like capability for learning algorithms, which naturally reduces the burden of labeling. In zero-shot learning problem, there are no training samples available for a set of classes, referred to as unseen classes. Instead, semantic information (in the form of visual attributes or textual features) is available for unseen classes (; 2014). Besides, we have standard supervised training data for a different set of classes, referred to as seen classes along with the semantic information of seen classes. The key to solving zero-shot learning problem is to leverage trained classifier on seen classes to predict unseen classes by transferring knowledge analogous to humans. Early variants of ZSL assume that during inference, samples are only from unseen classes. Recent observations; realize that such an assumption is not realistic. Generalized ZSL (GZSL) addresses this concern and considers a more practical variant. In GZSL there is no restriction on seen and unseen classes during inference. We are required to discriminate between all the classes. Clearly, GZSL is more challenging because the trained classifier is generally biased toward seen classes. In order to create a bridge between visual space and semantic attribute space, some methods utilize embedding techniques (; ; ; ; ; ; ; ; ; ; ;) and the others use semantic similarity between seen and unseen classes . Semantic similarity based models represent each unseen class as a mixture of seen classes. While the embedding based models follow three various directions; mapping visual space to semantic space (; ; ; ; ;), mapping semantic space to the visual space (; ;), and finding a latent space then mapping both visual and semantic space into the joint embedding space;;; ). The loss functions in embedding based models have training samples only from the seen classes. For unseen classes, we do not have any samples. It is not difficult to see that this lack of training samples biases the learning process towards seen classes only. One of the recently proposed techniques to address this issue is augmenting the loss function with some unsupervised regularization such as entropy minimization over the unseen classes. Another recent methodology which follows a different perspective is deploying Generative Adversarial Network (GAN) to generate synthetic samples for unseen classes by utilizing their attribute information; ). Although generative models boost the significantly, it is difficult to train these models. Furthermore, the training requires generation of large number of samples followed by training on a much larger augmented data which hurts their scalability. The two most recent state-of-the-art GZSL methods, CRnet and COSMO , both employ a complex mixture of experts approach. CRnet is based on k-means clustering with an expert module on each cluster (seen class) to map semantic space to visual space. The output of experts (cooperation modules) are integrated and finally sent to a complex loss (relation module) to make a decision. CRnet is a multi-module (multi-network) method that needs end-to-end training with many hyperparameters. Also COSMO is a complex gating model with three modules: a seen/unseen classifier and two expert classifiers over seen and unseen classes. Both of these methods have many modules, and hence, several hyperparameters; architectural, and learning decisions. A complex pipeline is susceptible to errors, for example, CRnet uses k-means clustering for training and determining the number of experts and a weak clustering will lead to bad . Our Contribution: We propose a simple fully connected neural network architecture with unified (both seen and unseen classes together) cross-entropy loss along with soft-labeling. Soft-labeling is the key novelty of our approach which enables the training data from the seen classes to also train the unseen class. We directly use attribute similarity information between the correct seen class and the unseen classes to create a soft unseen label for each training data. As a of soft labeling, training instances for seen classes also serve as soft training instance for the unseen class without increasing the training corpus. This soft labeling leads to implicit supervision for the unseen classes that eliminates the need for any unsupervised regularization such as entropy loss in. Soft-labeling along with crossentropy loss enables a simple MLP network to tackle GZSL problem. Our proposed model, which we call Soft-labeled ZSL (SZSL), is simple (unlike GANs) and efficient (unlike visual-semantic pairwise embedding models) approach which achieves the state-of-the-art performance in Generalized-ZSL setting on all five ZSL benchmark datasets and outperforms the state-of-the-art for some of them. In zero-shot learning problem, a set of training data on seen classes and a set of semantic information (attributes) on both seen and unseen classes are given. The training dataset includes n samples where x i is the visual feature vector of the i-th image and y i is the class label. All samples in D belong to seen classes S and during training there is no sample available from unseen classes U. The total number of classes is C = |S| + |U|. Semantic information or attributes a k ∈ R a, are given for all C classes and the collection of all attributes are represented by attribute matrix A ∈ R a×C. In the inference phase, our objective is to predict the correct classes (either seen or unseen) of the test dataset D. The classic ZSL setting assumes that all test samples in D belong to unseen classes U and tries to classify test samples only to unseen classes U. While in a more realistic setting i.e. GZSL, there is no such an assumption and we aim at classifying samples in D to either seen or unseen classes S ∪ U. 3 PROPOSED METHODOLOGY 3.1 NETWORK ARCHITECTURE As Figure 1 illustrates our architecture, We map visual space to semantic space, then compute the similarity score (dot-product) between true attributes and the attribute/semantic representation of the input (x). Finally, the similarity score is fed into a Softmax, and the probability of all classes are computed. For the visual features as the input, in all five benchmark datasets, we use the extracted visual features by a pre-trained ResNet-101 on ImageNet provided by. We do not fine-tune the CNN that generates the visual features unlike model in. In this sense, our proposed model is also fast and straightforward to train. In ZSL problem, we do not have any training instance from unseen classes, so the output nodes corresponding to unseen classes are always inactive during learning. Standard supervised training with cross entropy loss biases the network towards seen classes only. The true labels (hard labels) used for training only represent seen classes so the cross entropy cannot penalize unseen classes. Moreover, the available similarity information between the seen and unseen attributed is never utilized. We propose soft labeling based on the similarity between semantic attributes. For each seen sample, we represent its relationship to unseen categories by obtaining semantic similarity (dot-product) using the seen class attribute and all the unseen class attributes. In the simplest form, for every training data, we can find the nearest unseen class to the correct seen class label and assign a small probability q (partial membership or soft label) of this instance to be from the closest unseen class. Note, each training sample only contains a label which comes from the set of seen classes. With soft labeling, we enrich the label with partial assignments to unseen classes and as shows, soft labels act as a regularizer which allows each training case to enforce much more constraint on weights. In a more general soft labeling approach, we propose assigning a probability to all the unseen classes. A natural choice is to transform seen-to-unseen similarities to probabilities (soft labels) shown in Equation. The unseen distribution is obtained for each seen class by calculating dot-product of seen class attribute and all unseen classes attributes and squashing all these dot-product values by Softmax to acquire probabilities. In this case, we distribute the probability q among all unseen classes based on the obtained unseen distribution. This proposed strategy in a soft label for each seen image during training, which as we show later helps the network to learn unseen categories. In order to control the flatness of the unseen distribution, we utilize temperature parameter τ. Higher temperature in flatter distribution over unseen categories and lower temperature creates a more ragged distribution with peaks on nearest unseen classes. A small enough temperature basically in the nearest unseen approach. The Impact of temperature τ on unseen distribution is depicted in Figure 3.a for a particular seen class. Soft labeling implicitly introduces unseen visual features into the network without generating fake unseen samples as in generative methods; ). Hence our proposed approach is able to reproduce same effect as in generative models without the need to create fake samples and train generative models that are known to be difficult to train. Below is the formal description of temperature Softmax: where a i is the i-th column of attribute matrix A ∈ R a×C which includes both seen and unseen class attributes: And s i,j is the true similarity score between two classes i, j based on their attributes. τ and q are temperature parameter and total probability assigned to unseen distribution, respectively. Also y u i,k is the soft label (probability) of unseen class k for seen class i. It should be noted that q is the sum of all unseen soft labels i.e. k∈U y u i,k = q. The proposed method is a multi-class probabilistic classifier that produces a C-dimensional vector of class probabilities p for each sample x i as p(C-dimensional vector of all similarity scores of an input sample. The predicted similarity score between semantic representation of sample x i and attribute a k isŝ i,k g w (x i), a k. Each element of vector p, represents an individual class probability that can be shown below: This Softmax as the activation function of the last layer of the network is calculated on all classes. An established choice to train a multi-class probabilistic classifier is the cross-entropy loss which we later show naturally integrates our idea of soft labeling. Inspired by , in addition to the cross-entropy loss over soft targets, we also consider cross entropy-loss over true labels (hard labels) to improve the performance. During training, we aim at learning the nonlinear mapping g w i.e. obtaining network weights W through: where λ and γ are regularization factors which are obtained through hyperparameter tuning, and L(x i) is the weighted sum of cross-entropy loss over soft labels (L soft) and cross-entropy loss over hard labels (L hard) for each sample as shown below: where α ∈ is a hyperparameter. For better understanding, the hard-loss and soft-loss terms for each sample x i (or x for simplicity) are expanded and elaborated. The hard-loss term is a conventional cross-entropy loss L hard (x) = − C k=1 z k log(p k), where z k is the hard label. Clearly, hard-loss term alone does not work in ZSL regime since it does not penalize unseen classes. The soft-loss term is expanded to seen and unseen terms as follows: Utilizing Equation Hence the first two terms of L soft (x) is the weighted sum of cross-entropy of seen classes and crossentropy of unseen classes. In particular, first term penalizes and controls the relative (normalized) probabilities within all seen classes and the second term acts similarly within unseen classes. We also require to penalize the total probability of all seen classes (1 −q) and total probability of all unseen classes (q). This is accomplished through the last two terms of Equation which is basically a binary cross entropy loss. Intuitively soft-loss in Equation works by controlling the balance within seen/unseen classes (first two terms) as well as the balance between seen and unseen classes (last two terms). As we have shown in Equation, soft-loss enables the classifier to learn unseen classes by only being exposed to samples from seen classes. Hyperparameter q acts as a trade-off coefficient between seen and unseen cross-entropy losses (Figure 2). We can see that the regularizer is a weighted cross entropy on unseen class, which leverages similarity structure between attributes compared to uniform entropy function of DCN. DCN and all prior works use uniform entropy as regularizer, which does not capitalize on the known semantic similarity information between seen and unseen class attributes. At the inference time, our proposed SZSL method works the same as a conventional classifier, we only need to provide the test image and the network will produce class probabilities for all seen and unseen classes. We conduct comprehensive comparison of our proposed SZSL model with the state-of-the-art methods for GZSL settings on five benchmark datasets (Table 1). We present the detailed description of datasets in Appendix A. Our model outperforms the state-of-the-art methods on GZSL setting for all benchmark datasets. For the purpose of validation, we employ the validation splits provided along with the Proposed Split (PS) to perform cross-validation for hyper-parameter tuning. The main objective of GZSL is to simultaneously improve seen samples accuracy and unseen samples accuracy i.e. imposing a trade-off between these two metrics. As the , the standard GZSL evaluation metric is harmonic average of seen and unseen accuracy. This metric is chosen to encourage the network not be biased toward seen classes. Harmonic average of accuracies is defined as where A S and A U are seen and unseen accuracies, respectively. 11.3 74.6 19.6 8.0 73.9 14.4 3.7 55.7 6.9 23.5 59.2 33.6 14.7 30.5 19.8 ConSE 0.4 88.6 0.8 0.5 90.6 1.0 0.0 91.2 0.0 1.6 72.2 3.1 6.8 39.9 11.6 Sync 8.9 87.3 16.2 10.0 90.5 18.0 7.4 66.3 13.3 11.5 70.9 19.8 7.9 43.3 13.4 DeViSE 13.4 68.7 22.4 17.1 74.7 27.8 4.9 76.9 9.2 23.8 53.0 32.8 16.9 27.4 20.9 CMT 0.9 87.6 1.8 8.7 89.0 15.9 1.4 85.2 2.8 7.2 49.8 12.6 8.1 21.8 11.8 Generative Models f-CLSWGAN 57.9 61.4 59.6 ------43.7 57.7 49.7 42.6 36.6 39.4 SP-AEN 23.3 90.9 37.1 ---13.7 63.4 13.7 34.7 70.6 46.6 24.9 38.6 30.3 cycle-UWGAN 59.6 63.4 59. To evaluate SZSL, we follow the popular experimental framework and the Proposed Split (PS) in for splitting classes into seen and unseen classes to compare GZSL/ZSL methods. Utilizing PS ensures that none of the unseen classes have been used in the training of ResNet-101 on ImageNet. The input to the model is the visual features of each image sample extracted by a pre-trained ResNet-101 on ImageNet provided by. The dimension of visual features is 2048. We utilized Keras with TensorFlow back-end to implement our model. We used proposed unseen classes for validation (3-fold CV) and added 20% of train samples (seen classes) as seen validation samples to obtain GZSL validation sets. We crossvalidate τ ∈ [10 −2, 10], mini-batch size ∈ {64, 128, 256, 512, 1024}, q ∈, α ∈, hidden layer size ∈ {128, 256, 512, 1024, 1500} and activation function ∈{tanh, sigmoid, hard-sigmoid, relu} to tune our model. To obtain statistically consistent , the reported accuracies are averaged over 5 trials (using different initialization) after tuning hyper-parameters with cross-validation. Also we ran our experiments on a machine with 56 vCPU cores, Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHZ and 2 NVIDIA-Tesla P100 GPUs each with 16GB memory. The code is provided in the supplementary material. To demonstrate the effectiveness of SZSL model in GZSL setting, we comprehensively compare our proposed method with state-of-the-art GZSL models in Table 2. Since we use the standard proposed split, the published of other GZSL models are directly comparable. As reported in Table 2, accuracies of our model achieves the state-of-the-art GZSL performance on all five benchmark datasets and outperforms the state-of-the-art on AwA2 and aPY datasets. It is exciting and motivating while our architecture is much simpler compared to recently proposed CRnet and COSMO, yet, we achieve similar or better accuracies compared to them. We have only one simple fully connected neural network with 2 trainable layers, compared to CRnet with K mixture of experts followed by relation module with complex loss functions (pairwsie). Soft labeling employed in SZSL gives the model new flexibility to trade-off between seen and unseen accuracies during training and attain a higher value of harmonic accuracy A H, which is the standard metric for GZSL. Assigned unseen soft labels (unseen probability q) enables the classifier to gain more confidence in recognizing unseen classes, which in turn in considerably higher unseen accuracy A U. As the classifier is now discriminating between more classes we get marginally lower seen accuracy A S. However, balancing A S and A U with the cost of deteriorating A S leads to much higher A H. This trade-off phenomenon is depicted in Figure 2 for all datasets. The flexibility provided by soft labeling is examined by obtaining accuracies for different values of q. In Figure 2.a and 2.b, by increasing total unseen probability q, A U increases and A S decreases as expected. From the trade-off curves, there is an optimal q where A H takes its maximum value as shown in Figure 2. Maximizing A H is the primary objective in a GZSL problem that can be achieved by semantic similarity based soft labeling and the trade-off knob, q. It should be noted that both AwA and aPY datasets (Figure 2 .a and 2.b) are coarse-grained class datasets. In contrast, CUB and SUN datasets are fine-grained with hundreds of classes and highly unbalanced seen-unseen split, and hence their accuracies have different behavior concerning q, as shown in Figure 2.c and 2.d. However, harmonic average curve still has the same behavior and possesses a maximum value at an optimal q. We illustrate the intuition with AwA dataset , a ZSL benchmark dataset. Consider a seen class squirrel. We compute closest unseen classes to the class squirrel in terms of attributes. We naturally find that the closest class is rat and the second closest is bat, while other classes such as horse, dolphin, sheep, etc. are not close (Figure 3.a). This is not surprising as squirrel and rat share several attribute. It is naturally desirable to have a classifier that gives rat higher probability than other classes. If we force this softly, we can ensure that classifier is not blind towards unseen classes due to lack of any training example. From a learning perspective, without any regularization, we cannot hope classifier to classify unseen classes accurately. This problem was identified in, where they proposed entropy- based regularization in the form of Deep Calibration Network (DCN). DCN uses cross-entropy loss for seen classes, and regularize the model with entropy loss on unseen classes to train the network. Authors in DCN postulate that minimizing the uncertainty (entropy) of predicted unseen distribution of training samples, enables the network to become aware of unseen visual features. While minimizing uncertainty is a good choice of regularization, it does not eliminate the possibility of being confident about the wrong unseen class. Clearly in DCN's approach, for the above squirrel example, the uncertainty can be minimized even when the classifier gives high confidence to a wrong unseen class dolphin on an image of seen class squirrel. Utilizing similarity based soft-labeling implicitly regularizes the model in a supervised fashion. The similarity values naturally has information of how much certainty we want for specific unseen class. We believe that this supervised regularization is the critical difference why our model outperforms DCN with a significant margin. Figure 3 shows the effect of τ and the consequent assigned unseen distribution on accuracies for AwA dataset. Small τ enforces q to be concentrated on nearest unseen class, while large τ spread q over all the unseen classes and basically does not introduce helpful unseen class information to the classifier. The optimal value for τ is 0.2 for AwA dataset as depicted in Figure 3.b. The impact of τ on the assigned distribution for unseen classes is shown in Figure 3.a when seen class is squirrel in AwA dataset. Unseen distribution with τ = 0.2, well represents the similarities between seen class (squirrel) and similar unseen classes (rat, bat, bobcat) and basically verifies the of Figure 3.b where τ = 0.2 is the optimal temperature. While in the extreme cases, when τ = 0.01, distribution on unseen classes in mostly focused on the nearest unseen class, rat, and consequently the other unseen classes' similarities are ignored. Also τ = 10 flattens the unseen distribution which in high uncertainty and does not contribute helpful unseen class information to the learning. We proposed a discriminative GZSL classifier with visual-to-semantic mapping and cross-entropy loss. During training, while SZSL is trained on a seen class, it simultaneously learns similar unseen classes through soft labels based on semantic class attributes. We deploy similarity based soft labeling on unseen classes that allows us to learn both seen and unseen signatures simultaneously via a simple architecture. Our proposed soft-labeling strategy along with cross-entropy loss leads to a novel regularization via generalized similarity-based weighted cross-entropy loss that can successfully tackle GZSL problem. Soft-labeling offers a trade-off between seen and unseen accuracies and provides the capability to adjust these accuracies based on the particular application. We achieve state-of-the-art performance, in GZSL setting, on all five ZSL benchmark datasets while keeping the model simple, efficient and easy to train.

How to use cross-entropy loss for zero shot learning with soft labeling on unseen classes : a simple and effective solution that achieves state-of-the-art performance on five ZSL benchmark datasets.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR:

Question: In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress. In this work, we use a curriculum of progressively growing action spaces to accelerate learning. We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space. Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efficiently transfers data, value estimates, and state representations from restricted action spaces to the full task. We show the efficacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large, multi-agent action spaces. The value of curricula has been well established in machine learning, reinforcement learning, and in biological systems. When a desired behaviour is sufficiently complex, or the environment too unforgiving, it can be intractable to learn the behaviour from scratch through random exploration. Instead, by "starting small" , an agent can build skills, representations, and a dataset of meaningful experiences that allow it to accelerate its learning. Such curricula can drastically improve sample efficiency . Typically, curriculum learning uses a progression of tasks or environments. Simple tasks that provide meaningful feedback to random agents are used first, and some schedule is used to introduce more challenging tasks later during training . However, in many contexts neither the agent nor experimenter has such unimpeded control over the environment. In this work, we instead make use of curricula that are internal to the agent, simplifying the exploration problem without changing the environment. In particular, we grow the size of the action space of reinforcement learning agents over the course of training. At the beginning of training, our agents use a severely restricted action space. This helps exploration by guiding the agent towards rewards and meaningful experiences, and provides low variance updates during learning. The action space is then grown progressively. Eventually, using the most unrestricted action space, the agents are able to find superior policies. Each action space is a strict superset of the more restricted ones. This paradigm requires some domain knowledge to identify a suitable hierarchy of action spaces. However, such a hierarchy is often easy to find. Continuous action spaces can be discretised with increasing resolution. Similarly, curricula for coping with the large combinatorial action spaces induced by many agents can be obtained from the prior that nearby agents are more likely to need to coordinate. For example, in routing or traffic flow problems nearby agents or nodes may wish to adopt similar local policies to alleviate global congestion. Our method will be valuable when it is possible to identify a restricted action space in which random exploration leads to significantly more meaningful experiences than random exploration in the full action space. We propose an approach that uses off-policy reinforcement learning to improve sample efficiency in this type of curriculum learning. Since data from exploration using a restricted action space is still valid in the Markov Decision Processes (MDPs) corresponding to the less restricted action spaces, we can learn value functions in the less restricted action space with'off-action-space' data collected by exploring in the restricted action space. In our approach, we learn value functions corresponding to each level of restriction simultaneously. We can use the relationships of these value functions to each other to accelerate learning further, by using value estimates themselves as initialisations or as bootstrap targets for the less restricted action spaces, as well as sharing learned state representations. Empirically, we first demonstrate the efficacy of our approach in two simple control tasks, in which the resolution of discretised actions is progressively increased. We then tackle a more challenging set of problems with combinatorial action spaces, in the context of StarCraft micromanagement with large numbers of agents. Given the heuristic prior that nearby agents in a multiagent setting are likely to need to coordinate, we use hierarchical clustering to impose a restricted action space on the agents. Agents in a cluster are restricted to take the same action, but we progressively increase the number of groups that can act independently of one another over the course of training. Our method substantially improves sample efficiency on a number of tasks, outperforming learning any particular action space from scratch, a number of ablations, and an actor-critic baseline that learns a single value function for the behaviour policy, as in the work of. Code is available, but redacted here for anonymity. Curriculum learning has a long history, appearing at least as early as the work of in reinforcement learning, and for the training of neural networks since. In supervised learning, one typically has control of the order in which data is presented to the learning algorithm. For learning with deep neural networks, explored the use of curricula in computer vision and natural language processing. Many approaches use handcrafted schedules for task curricula, but others (; ;) study diagnostics that can be used to automate the choice of task mixtures throughout training. In a self-supervised control setting, use sensitivity analysis to automatically define a curriculum over action dimensions and prioritise their search space. In some reinforcement learning settings, it may also be possible to control the environment so as to induce a curriculum. With a resettable simulator, it is possible to use a sequence of progressively more challenging initial states . With a procedurally generated task, it is often possible to automatically tune the difficulty of the environments . Similar curricula also appear often in hierarchical reinforcement learning, where skills can be learned in comparatively easy settings and then composed in more complex ways later . use more general inter-task mappings to transfer Q-values between tasks that do not share state and action spaces. In adversarial settings, one may also induce a curriculum through self-play (; ;). In this case, the learning agents themselves define the changing part of the environment. A less invasive manipulation of the environment involves altering the reward function. Such reward shaping allows learning policies in an easier MDP, which can then be transferred to the more difficult sparse-reward task . It is also possible to learn reward shaping on simple tasks and transfer it to harder tasks in a curriculum . In contrast, learning with increasingly complex function approximators does not require any control of the environment. In reinforcement learning, this has often taken the form of adaptively growing the resolution of the state space considered by a piecewise constant discretised approximation (; ;). study continual complexification in the context of coevolution, growing the complexity of neural network architectures through the course of training. These works progressively increase the capabilities of the agent, but not with respect to its available actions. In the context of planning on-line with a model, there are a number of approaches that use progressive widening to consider increasing large action spaces over the course of search , including in planning for continuous action spaces (Couëtoux et al., 2011). However, these methods cannot directly be applied to grow the action space in the model-free setting. A recent related work tackling our domain is that of , who train mixtures of two policies with an actor-critic approach, learning a single value function for the current mixture of policies. The mixture contains a policy that may be harder to learn but has a higher performance ceiling, such as a policy with a larger action space as we consider in this work. The mixing coefficient is initialised to only support the simpler policy, and adapted via population based training. In contrast, we simultaneously learn a different value function for each policy, and exploit the properties of the optimal value functions to induce additional structure on our models. We further use these properties to construct a scheme for off-action-space learning which means our approach may be used in an off-policy setting. Empirically, in our settings, we find our approach to perform better and more consistently than an actor-critic algorithm modeled after , although we do not take on the significant additional computational requirements of population based training in any of our experiments. A number of other works address the problem of generalisation and representation for value functions with large discrete action spaces, without explicitly addressing the ing exploration problem . These approaches typically rely on action representations from prior knowledge. Such representations could be used in combination with our method to construct a hierarchy of action spaces with which to shape exploration. We formalise our problem as a MDP, specified by a tuple < S, A, P, r, γ >. The set of possible states and actions are given by S and A, P is the transition function that specifies the environment dynamics, and γ is a discount factor used to specify the discounted return R = T t=0 γ t r t for an episode of length T. We wish our agent to maximise this return in expectation by learning a policy π that maps states to actions. The state-action value function (Q-function) is given by The optimal Q-function Q * satisfies the Bellman optimality equation: Q-learning uses a sample-based approximation of the Bellman optimality operator T to iteratively improve an estimate of Q *. Q-learning is an off-policy method, meaning that samples from any policy may be used to improve the value function estimate. We use this property to engage Q-learning for off-action-space learning, as described in the next section. We also introduce some notation for restricted action spaces. In particular, for an MDP with unrestricted action space A we define a set of N action spaces A, ∈ {0, . . ., N − 1}. Each action space is a subset of the next: A 0 ⊂ A 1 ⊂... ⊂ A N −1 ⊆ A. A policy restricted to actions A is denoted π (a|s). The optimal policy in this restricted policy class is π * (a|s), and its corresponding action-value and value functions are Q * (s, a) and V * (s) = max a Q * (s, a). Additionally, we define a hierarchy of actions by identifying for every action a ∈ A, > 0 a parent action parent (a) in the space of A −1. Since action spaces are subsets of larger action spaces, for all a ∈ A −1, parent (a) = a, i.e., one child of each action is itself. Simple pieces of domain knowledge are often sufficient to define these hierarchies. For example, a discretised continuous action can identify its nearest neighbour in A −1 as a parent. In Section 5 we describe a possible hierarchy for multi-agent action spaces. One could also imagine using action-embeddings to learn such a hierarchy from data. We build our approach to growing action spaces (GAS) on off-policy value-based reinforcement learning. Q-learning and its deep-learning adaptations have shown strong performance , and admit a simple framework for off-policy learning. A value function for an action space A may be updated with transitions using actions drawn from its own action space, or any more restricted action spaces, if we use an off-policy learning algorithm. The restricted transitions simply form a subset of the data required to learn the value functions of the less restricted action spaces. To exploit this, we simultaneously learn an estimated optimal value functionQ * (s, a) for each action space A, and use samples drawn from a behaviour policy based on a value function for low to directly train the higher value functions. At the beginning of each episode, we sample according to some distribution. The experiences generated in that episode are used to update all of theQ * ≥ (s, a). This off-action-space learning is a type of off-policy learning that enables efficient exploration by restricting it to the low-regime. We sample at the beginning of the episode rather than at each timestep because, if the agent uses a high-action, it may enter a state that is inaccessible for a lower-policy, and we do not wish to force a low-value function to generalise to states that are only accessible at higher. Since data from a restricted action space only supports a subset of the state-action space relevant for the value functions of less restricted action spaces, we hope that a suitable function approximator still allows some generalisation to the unexplored parts of the less restricted state-action space. Note that: This is because each action space is a strict subset of the larger ones, so the agent can always in the worst case fall back to a policy using a more restricted action space. This monotonicity intuitively recommends an iterative decomposition of the value estimates, in whichQ * +1 (s, a) is estimated as a sum ofQ * (s, a) and some positive ∆ (s, a). This is not immediately possible due to the mismatch in the support of each function. However, we can leverage a hierarchical structure in the action spaces when present, as described in Section 3. In this case we can use: This is a task-specific upsampling of the lower-value function to intialise the next value function. BothQ * (s, a) and ∆ (s, a) are learned components. We could further regularise or restrict the functional form of ∆ to ensure its positivity when parent (a) = a. However, we did not find this to be valuable in our experiments, and simply initialised ∆ to be small. The property also implies a modified Bellman optimality equation: The max i< are redundant in their role as conditions on the optimal value function Q *. However, the Bellman optimality equation also gives us the form of a Q-learning update, where the term in the expectation on the RHS is used as an operator that iteratively improves an estimate of Q *. When these estimates are inaccurate, the modified form of the Bellman equation may lead to different updates, allowing the solutions to higher to be bootstrapped from those at lower. We expect that policies with low are easier to learn, and that therefore the correspondingQ * is higher value and more accurate earlier in training than those at high. These high values could be picked up by the extra maximisation in the modified bootstrap, and thereby rapidly learned by the higher-value functions. Empirically however, we find that using this form for the target in our loss function performs no better than just maximising overQ * (s, a). We discuss the choice of target and these in more detail in Section 6.2. By sharing parameters between the function approximators of each Q, we can learn a joint state representation, which can then be iteratively decoded into estimates of Q * for each. This shared embedding can be iteratively refined by, e.g., additional network layers for eachQ * to maintain flexibility along with transfer of useful representations. This simple approach has had great success in improving the efficiency of many multi-task solutions using deep learning . We need to choose a schedule with which to increase the used by the behaviour policy over the course of training. use population based training to choose a mixing parameter on the fly. However, this comes at significant computational cost, and optimises greedily for immediate performance gains. We use a simple linear schedule on a mixing parameter α ∈ [0, N]. Initially α = 0 and we always choose = 0. Later, we pick = α with probability α − α and = α with probability α − α (e.g. if α = 1.1, we choose = 1 with 90% chance and = 2 with 10% chance). This worked well empirically with little effort to tune. Many other strategies exist for tuning a curriculum automatically (such as those explored by), and could be beneficial, at the cost of additional overhead and algorithmic complexity. In cooperative multi-agent control, the full action space allows each of N agents to take actions from a set A agent, ing in an exponentially large action space of size |A agent | N. Random exploration in this action space is highly unlikely to produce sensical behaviours, so growing the action space as we propose is particularly valuable in this setting. One approach would be to limit the actions available to each agent, as done in our discretised continuous control experiments (Section 6.1) and those of. However, the joint action space would still be exponential in N. We propose instead to use hierarchical clustering, and to assign the same action to nearby agents. At the first level of the hierarchy, we treat the whole team as a single group, and all agents are constrained to take the same action. At the next level of the hierarchy, we split the agents into k groups using an unsupervised clustering algorithm, allowing each group to act independently. At each further level, every group is split once again into k smaller groups. In practice, we simply use k-means clustering based on the agent's spatial position, but this can be easily extended to more complex hierarchies using other clustering approaches. To estimate the value function, we compute a state-value scoreV (s), and a group-action delta ∆ (s, a g, g) for each group g at each level. Then, we compute an estimated group-action value for each group, at each level, using a per-group form of. We useQ * −1 (s, ·) =V (s) to initialise the iterative computation, similarly to the dueling architecture of. The estimated value of the parent action is the estimated value of the entire parent group all taking the same action as the child group. At each level we now have a set of group-action values. In effect, a multi-agent value-learning problem still remains at each level, but with a greatly reduced number of agents at low. We could simply use independent Q-learning , but instead choose to estimate the joint-action value at each level as the mean of the group-action values for the groups at that, as in the work of. A less restrictive representation, such as that proposed by , could help, but we leave this direction to future work. A potential problem is that the clustering changes for every state, which may interfere with generalisation as group-actions will not have consistent semantics. We address this in two ways. First, we include the clustering as part of the state, and the cluster centroids are re-initialised from the previous timestep for t > 0 to keep the cluster semantics approximately consistent. Second, we use a functional representation that produces group-action values that are broadly agnostic to the identifier of the group. In particular, we compute a spatially resolved embedding, and pool over the locations occupied by each group. See Figure 2 and Section 6.2 for more details. We investigate two classes of problems that have a natural hierarchy in the action space. First, simple control problems where a coarse action discretisation can help accelerate exploration, and fine action discretisation allows for a more optimal policy. Second, the cooperative multi-agent setting, discussed in Section 5, using large-scale StarCraft micromanagement scenarios. As a proof-of-concept, we look at two simple examples: versions of the classic Acrobot and Mountain Car environments with discretised action spaces. Both tasks have a sparse reward of +1 when the goal is reached, and we make the exploration problem more challenging by terminating episodes Figure 1: Discretised continuous control with growing action spaces. We report the mean and standard error (over 10 random seeds) of the returns during training, with a moving average over the past 20 episodes. A 2 (slow) is an ablation of A 2 that decays at a quarter the rate. with a penalty of -1 if the goal is not reached within 500 timesteps. The normalised remaining time is concatenated to the state so it remains Markovian despite the time limit. There is a further actuation cost of 0.05 a 2. At A 0, the actions apply a force of +1 and −1. At each subsequent A >0, each action is split into two children, one that is the same as the parent action, and the other applying half the force. Thus, there are 2 actions in A. The of our experiments are shown in Figure 1. Training with the lower resolutions A 0 and A 1 from scratch converges to finding the goal, but incurs significant actuation costs. Training with A 2 from scratch almost never finds the goal with -greedy exploration. We also tried decaying the at a quarter of the rate (A 2 slow) without success. In these cases, the policy converges to the one that minimises actuation costs, never finding the goal. Training with a growing action space explores to find the goal early, and then uses this experience to transition smoothly into a solution that finds the goal but takes a slower route that minimises actuation costs while achieving the objective. The real-time strategy game StarCraft and its sequel StarCraft II have emerged as popular platforms for benchmarking reinforcement learning algorithms. Full game-play has been tackled by e.g. , while other works focus on sub-problems such as micromanagement, the low-level control of units engaged in a battle between two armies (e.g.). Efforts to approach the former problem have required some subset of human demonstrations, hierarchical methods, and massive compute scale, and so we focus on the latter as a more tractable benchmark to evaluate our methods. Most previous work on RL benchmarking with StarCraft micromanagement is restricted to maximally 20-30 units (; . In our experiments we focus on much larger-scale micromanagement scenarios with 50-100 units on each side of the battle. To further increase the difficulty of these micromanagement scenarios, in our setting the starting locations of the armies are randomised, and the opponent is controlled by scripted logic that holds its position until any agent-controlled unit is in range, and then focus-fires on the closest enemy. This increases the exploration challenge, as our agents need to learn to find the enemy first, while they hold a strong defensive position. The action space for each unit permits an attack-move or move action in eight cardinal directions, as well as a stop action that causes the unit to passively hold its position. In our experiments, we use k = 2 for k-means clustering and split down to at most four or eight groups. The maximum number of groups in an experiment with A is 2 . Although our approach is designed for off-policy learning, we follow the common practice of using n-step Q-learning to accelerate the propagation of values . Our base algorithm uses the objective of n-step Q-learning from the work of , and collects data from multiple workers into a short queue similarly to. Full details can be found in the Appendix. We propose an architecture to efficiently represent the value functions of the action-space hierarchy. The overall structure is shown in Figure 2. We start with the state of the scenario. Ally units are blue and split into two groups. From the state, features are extracted from the units and map (see Appendix for full details). These features are concatenated with a one-hot representation of the unit's group (for allied agents), and are embedded with a small MLP. A 2-D grid of embeddings is constructed by adding up the unit embeddings for all units in each cell of the grid. The embeddings are passed through a residual CNN to produce a final embedding, which is copied several times and decoded as follows. First, a state-value branch computes a scalar value by taking a global mean pooling and passing the through a 2-layer MLP. Then, for each, a masked mean-pooling is used to produce an embedding for each group at that A by masking out the positions in the spatial embedding where there are no units of that group (5a, 5b, 5c). A single evaluation MLP for each is used to decode this embedding into a group action-score (7a, 7b, 7c). This architecture allows a shared state representation to be efficiently decoded into value-function contributions for groups of any size, at any level of restriction in the action space. We consider two approaches for combining these outputs. In our default approach, described in Section 5, each group's action-value is given by the sum of the state-value and group-action-scores for the group and its parents (8a, 8b). In'SEP-Q', each group's action-value is simply given by the state-value added to the group-action score, i.e.,Q * (s, a g) =V (s) + ∆ (s, a g, g). This is an ablation in which the action-value estimates for restricted action spaces do not initialise the actionvalue estimates of their child actions. Figure 3 presents the of our method, as well as a number of baselines and ablations, on a variety of micromanagement tasks. Our method is labeled Growing Action Spaces GAS, such that GAS will grow from A 0 to A 2. Our primary baselines are policies trained with action spaces A 0 or A 2 from scratch. GAS consistently outperforms both of these variants. Policies trained from scratch on A 2 struggle with exploration, in particular in the harder scenarios where the opponent has a numbers advantage. Policies trained from scratch on A 0 learn quickly, but plateau comparatively low, due to the limited ability of a single group to position effectively. GAS benefits from the efficient exploration enabled by an intialisation at A 0, and uses the data gathered under this policy to efficiently transfer to A 2; enabling a higher asymptotic performance. We also compare against a Mix&Match (MM) baseline using the actor-critic approach of , but adapted for our new multi-agent setting and supporting a third level in the mixture Figure 3: StarCraft micromanagement with growing action spaces. We report the mean and standard error (over 5 random seeds) of the evaluation winrate during training, with a moving average over the past 500 episodes. of policies (A 0, A 1, A 2). We tuned hyperparameters for all algorithms on the easiest, fastesttraining scenario (80 marines vs. 80 marines). On this scenario, MM learns faster but plateaus at the same level as GAS. MM underperforms on all other scenarios to varying degrees. Learning separate value functions for each A, as in our approach, appears to accelerate the transfer learning in the majority of settings. Another possible explanation is that MM may be more sensitive to hyperparameters. We do not use population based training to tune hyperparameters on the fly, which could otherwise help MM adapt to each scenario. However, GAS would presumably also benefit from population based training, at the cost of further computation and sample efficiency. The policies learned by GAS exhibit good tactics. Control of separate groups is used to position our army so as to maximise the number of attacking units by forming a wall or a concave that surrounds the enemy, and by coordinating a simultaneous assault. Figure 5 in the Appendix shows some example learned policies. In scenarios where MM fails to learn well, it typically falls into a local minimum of attacking head-on. In each scenario, we test an ablation GAS: ON-AC that does not use our off-action-space update, instead training each level of the Q-function only with data sampled at that level. This ablation performs somewhat worse on average, although the size of the impact varies in different scenarios. In some tasks, it is beneficial to accelerate learning for finer action spaces using data drawn from the off-action-space policy. In Appendix A.1.1, the same ablation shows significantly worse performance on the Mountain Car task and comparable performance on Acrobot. We present a number of further ablations on two scenarios. The most striking failure is of the'SEP-Q' variant which does not compose the value function as a sum of scores in the hierarchy. It is critical to ensure that values are well-initialised as we move to less restricted action spaces. In the discretised continuous control tasks,'SEP-Q' also underperforms, although less dramatically. The choice of target is less important: performing a max over coarser action spaces to construct the target as described in Section 4.2 does not improve learning speed as intended. One potential reason is that maximising over more potential targets increases the maximisation bias already present in Q-learning . Additionally, we use an n-step objective which combines a partial onpolicy return with the bootstrap target, which could reduce the relative impact of the choice of target. Finally, we experiment with a higher. Unfortunately, asymptotic performance is degraded slightly once we use A 3 or higher. One potential reason is that it decreases the average group size, pushing against the limits of the spatial resolution that may be captured by our CNN architecture. Higher increases the amount of time that there are fewer units than groups, leaving certain groups empty and rendering our masked pooling operation degenerate. We do not see a fundamental limitation that should restrict the further growth of the action space, although we note that most hierarchical approaches in the literature avoid too many levels of depth. For example, only mix between two sizes of action spaces rather than the three we progress through in the majority of our GAS experiments. In this work, we presented an algorithm for growing action spaces with off-policy reinforcement learning to efficiently shape exploration. We learn value functions for all levels of a hierarchy of restricted action spaces simultaneously, and transfer data, value estimates, and representations from more restricted to less restricted action spaces. We also present a strategy for using this approach in cooperative multi-agent control. In discretised continuous control tasks and challenging multiagent StarCraft micromanagement scenarios, we demonstrate empirically the effectiveness of our approach and the value of off-action-space learning. An interesting avenue for future work is to automatically identify how to restrict action spaces for efficient exploration, potentially through meta-optimisation. We also look to explore more complex and deeper hierarchies of action spaces. A.1 DISCRETISED CONTINUOUS CONTROL A.1.1 ADDITIONAL ABLATIONS Here, we present on some additional ablations of GAS on the discretised continous control tasks. SEP-Q performs slightly worse on both tasks, a less dramatic failure than in the StarCraft experiments. These value functions are simpler, and it is easier to learn the new action space's value without relying so much on the previous one. ON-AC performs worse only on Mountain Car, suggesting once again that the significance of this component of the algorithm is somewhat problemdependent. We also test a version that follows the intuition of the'Match' objective of M&M more closely, adapted for the value-based setting: instead of using an adaptive initialisation of each level's Q-function as described in the main text, we use an L2 penalty to'Match' the new level's value function to its parent action, which should have a similar effect. This variant performs similarly here (perhaps slightly worse in the more challenging Mountain Car). For our experiments in discretised continous control, we use a standard DQN trainer with the following parameters. For GAS experiments, we keep the mixing coefficient α = 0 for 25000 environment steps, and then increase it linearly by 1 every 25000 steps until reaching the maximum value. We use γ = 0.998 for our Acrobot experiments, but reduce it to γ = 0.99 for Mountain Car to prevent diverging Q-values. Our model consists of fully-connected ReLU layers, with 128 hidden units for the first and 64 hidden units for all subsequent layers. Two layers are applied as an encoder. Then, for each one layer is applied on the current embedding to produce a new embedding, and an evaluation layer on that embedding produces the Q-values for that level. In these examples, the opponent is always on the right, and the agent controlled by model trained with GAS is on the left. We explore five Starcraft micromanagement scenarios: 50 hydralisks vs 50 hydralisks, 80 marines vs 80 marines, 80 marines vs 85 marines, 60 marines vs 65 marines, 95 zerglings vs 50 marines. In these scenarios, our model controls the first set of units, and the opponent controls the second set. The opponent is a scripted opponent that holds its location until an opposing unit is within range to attack. Then, the opponent will engage in an "attack-closest" behavior, as described in, where each unit individually targets the closest unit to it. Having the opponent remain stationary until engaged makes this a more difficult problem -the agent must find its opponent, and attack into a defensive position, which requires good positions prior to engagement. As mentioned in section 6.2, all of our scenarios require control of a much larger number of units than previous work. The 50 hydralisks and 80v80 marines scenarios are both imbalanced as a of attacking into a defensive position. The optimal strategy for 80 marines vs 85 marines and 60 vs 65 marines requires slightly more sophisticated unit positioning, and the 95 zerglings vs 50 marines scenario requires the most precise positioning. The agent can use the enemy's initial stationary positioning to its advantage by slightly surrounding the opponent in a concave, ensuring that the outermost units are in its attack range, but far enough away to be out of range of the center-most enemy units. Ideally, the timing of the groups in all scenarios should be coordinated such that all units get in range of the opponent at roughly the same point in time. Figure 5 shows how our model is able to exhibit this level of unit control. We use a standard features for the units and map, given by TorchcraftAI 1 For each of the units, the following features are extracted: 1 https://github.com/TorchCraft/TorchCraftAI • Current x, y positions. • Current x, y velocities. • Current hitpoints • Armor and damage values • Armor and damage types • Range versus both ground and air units • Current weapon cooldown • A few boolean flags on some miscellaneous unit attributes Approximate normalization for each feature keep its value approximately between 0-1. For the map, the following features are extracted for each tile in the map: • a one-hot encoding of tile's the ground height (4 channels) • boolean representing or not the given tile is walkable • boolean representing or not the given tile is buildable • and boolean representing or not the given tile is covered by fog of war. The features form a HxW x7 tensor, where our map has height H and width W. We use a frame-skip of 25, approximately 1 second of real time, allowing for reasonably fine-grained control but without making the exploration and credit assignment problems too challenging. We calculate at every timestep the difference in total health points (HP) and number of units for the enemy from the last step, normalised by the total starting HP and unit count. As a reward function, we use the normalised damage dealt, plus 4 times the normalised units killed, plus an additional reward of 8 for winning the scenario by killing all enemy units. This reward function is designed such that the agent gets some reward for doing damage and killing units, but the reward from doing damage will never be greater than from winning the scenario. Ties and timeouts are considered losses. A.3.1 MODEL As described in Section 6.2.2 a custom model architecture is used for Starcraft micromanagement. Each unit's feature vector is embedded to size 128 in step 2 of Figure 2. The grid where the unit features and map features are scattered onto is the size of the Starcraft map of the scenario in walktiles downsampled by a factor of 8. After being embedded, the unit features for ally and enemy units are concatenated with the downsampled map features and sent into a ResNet encoder with four residual blocks (stride 7 padding 3). The output is an embedding of size 64. The decoder uses a mean pooling over the embedding cells as described in Section 6.2.2. Each evaluator is a 2-layer MLP with 64 hidden units and 17 outputs, one for each action. All layers are separated with ReLU nonlinearities. We use 64 parallel actors to collect data in a short queue from which batches are removed when they are consumed by the learner. We use batches of 32 6-step segments for each update. For the Q-learning experiments, we used the Adam optimizer with a learning rate of 2.5 × 10 −4 and = 1 × 10 −4. For the MM baseline experiments, we use a learning rate of 1 × 10 −4, entropy loss coefficient of 8 × 10 −3 and value loss coefficient 0.5. The learning rates and entropy loss coefficient were tuned by random search, training with A 0 from scratch on the 80 marines vs 80 marines scenario with 10 configurations sampled from log uniform(−5, −3) for the learning rate and log uniform(−3, −1) for the entropy loss coefficient. For Q-learning, we use an -greedy exploration strategy, decaying linearly from 1.0 to 0.1 over the first 10000 model updates. We also use a target network that copies the behaviour model's parameters every 200 model updates. We also use a linear schedule to grow the action-space. There is a lead in of 5000 model updates, during which the action-space is held constant at A 0, to prevent the action space from growing when or the policy entropy is too high. The action-space is then grown linearly at a rate of 10000 model updates per level of restriction, so that after 10000 updates, we act entirely at A 1 and after 20000, entirely at A 2.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Progressively growing the available action space is a great curriculum for learning agents

Represent the TLDR for finding the corresponding scientific article

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the scientific article to find a short TLDR

Question: Recently, researchers have discovered that the state-of-the-art object classifiers can be fooled easily by small perturbations in the input unnoticeable to human eyes. It is known that an attacker can generate strong adversarial examples if she knows the classifier parameters. Conversely, a defender can robustify the classifier by retraining if she has the adversarial examples. The cat-and-mouse game nature of attacks and defenses raises the question of the presence of equilibria in the dynamics. In this paper, we present a neural-network based attack class to approximate a larger but intractable class of attacks, and formulate the attacker-defender interaction as a zero-sum leader-follower game. We present sensitivity-penalized optimization algorithms to find minimax solutions, which are the best worst-case defenses against whitebox attacks. Advantages of the learning-based attacks and defenses compared to gradient-based attacks and defenses are demonstrated with MNIST and CIFAR-10. Recently, researchers have made an unsettling discovery that the state-of-the-art object classifiers can be fooled easily by small perturbations in the input unnoticeable to human eyes BID24 BID7. Following studies tried to explain the cause of the seeming failure of deep learning toward such adversarial examples. The vulnerability was ascribed to linearity BID24, low flexibility BID4, or the flatness/curvedness of decision boundaries BID20, but a more complete picture is still under research. This is troublesome since such a vulnerability can be exploited in critical situations such as an autonomous car misreading traffic signs or a facial recognition system granting access to an impersonator without being noticed. Several methods of generating adversarial examples were proposed BID7 BID19 BID2, most of which use the knowledge of the classifier to craft examples. In response, a few defense methods were proposed: retraining target classifiers with adversarial examples called adversarial training BID24 BID7; suppressing gradient by retraining with soft labels called defensive distillation BID22; hardening target classifiers by training with an ensemble of adversarial examples BID25.In this paper we focus on whitebox attacks, that is, the model and the parameters of the classifier are known to the attacker. This requires a more robust classifier or defense method than simply relying on the secrecy of the parameters as defense. When the classifier parameters are known to an attacker, existing attack methods are very successful at fooling the classifiers. Conversely, when the attack is known to the classifier, e.g., in the form of adversarial examples, one can weaken the attack by retraining the classifier with adversarial examples, called adversarial training. However, if we repeat adversarial sample generation and adversarial training back-to-back, it is observed that the current adversarially-trained classifier is no longer robust to previous attacks (see Sec. 3.1.) To find the classifier robust against the class of gradient-based attacks, we first propose a sensitivitypenalized optimization procedure. Experiments show that the classifier from the procedure is more robust than adversarially-trained classifiers against previous attacks, but it still remains vulnerable to some degrees. This raises the main question of the paper: Can a classifier be robust to all types of attacks? The answer seems to be negative in light of the strong adversarial examples that can be crafted by direct optimization procedures from or BID2. Note that the class of optimization-based attack is very large, as there is no restriction on the adversarial patterns that can be generated except for certain bounds such as l p -norm bounds. The vastness of the optimization-based attack class is a hindrance to the study of the problem, as the defender cannot learn efficiently about the attack class from a finite number of samples. To study the problem analytically, we use a class of learning-based attack that can be generated by a class of neural networks. This class of attack can be considered an approximation of the class of optimization -based attacks, in that the search space of optimal perturbation is restricted to the parameter space of a neural network architecture, e.g., all perturbations that can be generated by fully-connected 3-layer ReLU networks. Similar to what we propose, others have recently considered training neural networks to generate adversarial examples BID21 BID0. While the proposed learning-based attack is weaker than the optimization-based attack, it can generate adversarial examples in test time with only single feedforward passes, which makes real-time attacks possible. We also show that the class of neural-network based attacks is quite different from the the class of gradient-based attacks (see Sec. 4.1.) Using the learning-based attack class, we introduce a continuous game formulation for analyzing the dynamics of attack-defense. The game is played by an attacker and a defender/classifier 1, where the attacker tries to maximize the risk of the classification task by perturbing input samples under certain constraints such as l p -norm bounds, and the defender/classifier tries to adjust its parameters to minimize the same risk given the perturbed inputs. It is important to note that for adversarial attack problems, the performance of an attack or a defense cannot be measured in isolation, but only in pairs of (attack, defense). This is because the effectiveness of an attack/defense depends on the defense/attack it is against. As a two-player game, there may not be a dominant defense that is no less robust than all other defenses against all attacks. However, there is a natural notion of the best defense or attack in the worst case. Suppose one player moves first by choosing her parameters and the other player responds with the knowledge of the first player's move. This is an example of a leader-follower game BID1 for which there are two well-known states, the minimax and the maximin solutions if it is a constant-sum game. To find those solutions empirically, we propose a new continuous optimization method using the sensitivity penalization term. We show that the minimax solution from the proposed method is indeed different from the solution from the conventional alternating descent/ascent and is also more robust. We also show that the strength/weakness of the minimax-trained classifier is different from that of adversarially-trained classifiers for gradient-based attacks. The contributions of this paper are summarized as follows.• We provide a continuous game model to analyze adversarial example attacks and defenses, using the neural network-based attack class as a feasible approximation to a larger but intractable class of optimization-based attacks.• We demonstrate the difficulty of defending against multiple attack types and present the minimax defense as the best worst-case defense methods.• We propose a sensitivity-penalized optimization method (Alg. 1) to numerically find continuous minimax solutions, which is better than alternating descent/ascent. The proposed optimization method can also be used for other minimax problems beyond the adversarial example problem. The proposed methods are demonstrated with the MNIST and the CIFAR-10 datasets. For readability, details about experimental settings and the with CIFAR-10 are presented in the appendix. Making a classifier robust to test-time adversarial attacks has been studied for linear (kernel) hyperplanes BID13, naive Bayes BID3 and SVM BID5, which also showed the game-theoretic nature of the robust classification problems. Since the recent discovery of adversarial examples for deep neural networks, several methods of generating adversarial samples were proposed BID24 BID7 BID19 BID2 as well as several methods of defense BID24 BID7 BID22 BID25. These papers considered static scenarios, where the attack/defense is constructed against a fixed opponent. A few researchers have also proposed using a detector to detect and reject adversarial examples BID16 BID14 BID18. While we do not use detectors in this work, the minimax approach we proposed in the paper can be applied to train the detectors. The idea of using neural networks to generate adversarial samples has appeared concurrently BID0 BID21. Similar to our paper, the two papers demonstrates that it is possible to generate strong adversarial samples by a learning approach. BID0 explored different architectures for the "adversarial transformation networks" against several different classifiers. BID21 proposed "attack learning neural networks" to map clean samples to a region in the feature space where misclassification occurs and "defense learning neural networks" to map them back to the safe region. Instead of prepending the defense layers before the fixed classifier BID21, we retrain the whole classifier as a defense method. However, the key difference of our work to the two papers is that we consider the dynamics of a learning-based defense stacked with a learning-based attack, and the numerical computation of the optimal defense/attack by continuous optimization. The alternating gradient-descent method for finding an equilibrium of a game has gained renewed interest since the introduction of Generative Adversarial Networks (GAN) BID6. However, the instability of the alternating gradient-descent method has been known, and the "unrolling" method BID17 was proposed to speed up the GAN training. The optimization algorithm proposed in the paper has a similarity with the unrolling method, but it is simpler (corresponding to a single-step unrolling) and involves a gradient-norm regularization which can be interpreted intuitively as sensitivity penalization BID8 BID15. Lastly, the framework of minimax risks was also studied in BID9 for the purpose of privacy preservation. We propose a different algorithm in this paper, but we also show that the attack on classification and the attack on privacy are the two sides of the same optimization problem with the opposite goals.3 CAT-AND-MOUSE GAME A classifier whose parameters are known to an attacker is easy to attack. Conversely, an attacker whose sample-generating method is known to a classifier is easy to defend from. In this section, we demonstrate the cat-and-mouse nature of the interaction, using adversarial training (Adv Train) as defense and the fast gradient sign method (FGSM) BID7 and the iterative version (IFGSM) BID11 as attacks. We then show that the equilibrium, if it exists, can be found more efficiently by directly solving a sensitivity-penalized optimization problem. Suppose g is a classifier g: X → Y and l(g(x), y) is a loss function. The FGSM attack generates a perturbed example z(x) given the clean sample x as follows: DISPLAYFORM0 The clean input images we use here are l ∞ -normalized, that is, all pixel values are in the range [−1, 1]. It was argued that the use of true label y in "label leaking" BID12 ), but we use will true labels in the paper for simplicity. For another attack example, the IFGSM attack iteratively refines an adversarial example by the following update DISPLAYFORM1 where the clipping used in this paper is clip x,η (x) min{1, x + η, max{−1, x − η, x}}.Existing attack methods such as FGSM and IFGSM are very effective at fooling the classifier. Table 1 shows that the two methods are able to perfectly fool a convolutional neural network trained with clean images from MNIST. (Details of the classifier architecture and the settings are in the appendix.)On the other hand, these attacks, if known to the classifier, can be weakened by retraining the classifier with the original dataset augmented by adversarial examples with ground-truth labels, known as adversarial training. In this paper we use the 1:1 mixture of the clean and the adversarial samples for adversarial training. Table 2 shows the of adversarial training for different attacks. Defense\Attack No attack FGSM IFGSM η=0.3 η=0.4 η=0.5 η=0.6 η=0.3 η=0.4 η=0.5 η=0.6 No defense 0.006 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 Table 1: Test error rates of FGSM and IFGSM attacks on an undefended convolutional neural network for MNIST. These attacks can cause perfect misclassification for the given range of η. The test error rates for adversarial test examples after training become below 1% indicating nearperfect avoidance. This is in stark contrast with the perfect misclassification of the undefended classifier in Table 1.Defense\Attack No attack FGSM IFGSM η=0.3 η=0.4 η=0.5 η=0.6 η=0.3 η=0.4 η=0.5 η=0.6 Adv train n/a 0.004 0.003 0.003 0.005 0.003 0.003 0.004 0.010 Table 2: Error rates of FGSM and IFGSM attacks on adversarially-trained classifiers for MNIST. This defense can avert the attacks and achieve the error rates of the no-attack case. A question arises as to what would happen if the procedure of 1) adversarial sample generation using the current classifier, and 2) retraining classifier using the current adversarial examples is repeated for many rounds. The answer to this cat-and-mouse game is easy to experiment although time-consuming. Let's denote the attack on the original classifier as FGSM1, and the corresponding retrained classifier as Adv FGSM1. Repeating the procedure above generates the sequence of models FGSM1 → Adv FGSM1 → FGSM2 → Adv FGSM2, etc. FIG0 shows one such trial with 80 + 80 rounds of the procedure. Initially, the attacker achieves near-perfect attacks (i.e., error rate 1), and the defender achieves near-perfect defense (i.e., error rate 0). As the iteration increases, the attacker becomes weaker with error rate 0.5, but the defense is still very successful, and the rate seems to oscillate persistently. While we can run more iterations to see if it converges, this is not a very principled nor efficient approach to find an equilibrium, if it exists. We can perform the cat-and-mouse simulation more efficiently by an optimization approach. Instead of training the classifier fully with adversarial examples and then regenerating adversarial examples, suppose we only update the classifier with a single gradient-descent step then regenerate adversarial examples. To emphasize the parameters u of the classifier/defender g(x; u), let's rewrite the empirical risk of classifying the perturbed data as DISPLAYFORM0 where z(x) denote an FGSM-like attack based on the loss gradient DISPLAYFORM1 and DISPLAYFORM2 ) is the sequence of perturbed examples. In expectation of the attack, the defender should choose u to minimize f (u, Z(u)) where the dependence of the attack on the classifier u is expressed explicitly. If we minimize f using gradient descent DISPLAYFORM3 then from the chain rule, the total derivative DISPLAYFORM4 from FORMULA2 and.Interestingly, this total derivative at the current state coincides with the gradient ∇ u of the following cost DISPLAYFORM5 where γ = ηN. There are two implications. Interpretation-wise, this cost function is the sum of the original risk f and the'sensitivity' term ∂f /∂Z 2 which penalizes abrupt changes of the risk w.r.t. the input. Therefore, u is chosen at each iteration to not only decrease the risk but also to make the classifier insensitive to input perturbation so that the attacker cannot take advantage of large gradients. The idea of minimizing the sensitivity to input is a familiar approach in robustifying classifiers BID8 BID15. Secondly, the new formulation can be implemented easily. The gradient descent update using the seemingly complicated gradient can be replaced by the gradient descent update of. The capability of automatic differentiation BID23 in modern machine learning libraries can be used to compute the gradient of FORMULA7 efficiently. Using this direct approach, we can find the defense parameters u which will be robust to gradient-based attacks. Fig. 2 shows the decrease of test error during training using the this gradient descent approach for MNIST. It only takes a very small fraction of time to reach the final states of the There is also an important difference between the solution of the cat-and-mouse game and the minimizer of. Table 3 shows that the adversarially trained classifier (Adv FGSM1) is robust to both clean data and FGSM1 attack, but is susceptible to FGSM2 attack, displaying the cat-and-mouse nature. The same holds for Adv FGSM2, Adv FGSM3, etc. After 80 rounds of the cat-and-mouse procedure, the classifier Adv FGSM80 becomes robust to FGSM80 as well as moderately robust to other attacks including FGSM81 (=FGSM-curr). However, the classifier Sens FGSM from direct minimization of FORMULA7 is even more robust toward FGSM-curr than Adv FGSM80 and is overall the best. To see the advantage of the sensitivity term in FORMULA7, we also performed the minimization of FORMULA7 without the sensitivity term under the same conditions as Sens FGSM. This optimization method is similar to the method proposed in, referred to as Learning with Adversaries (LWA FGSM). In the table, one can see that Sens FGSM is also better than LWA FGSM overall, although the difference is small. Note that Sens FGSM is better than other adversarially-trained classifiers, it too is still vulnerable to attacks such as FGSM80. This vulnerability raises the question if it is possible to make a classifier robust to any type of attacks, or more practically, robust to at least a large class of attacks. We discuss this issue in the next section. Table 3: Error rates of different attacks on various adversarially-trained classifiers for MNIST. FGSM-curr means the FGSM attack on the specific classifier on the left. Adv FGSM is the classifier adversarially trained with FGSM attacks. Sens FGSM is the of minimizing FORMULA7 by gradient descent. LWA FGSM is the of minimizing FORMULA7 without the gradient-norm term. In this section, we consider the class of optimization-based attack and the class of neural-network based attacks as an approximation of the former. Using the neural-network based attack class, we formulate the attacker-defender dynamics as a game and discuss two types of equilibria -the minimax and the maximin solutions. We present algorithms that generalize the approach presented in the previous section. An attacker z(x): X → X can be more general than a specific class of attacks such as FGSM. Again, let g: X → Y is a classifier parameterized by u and l(g(x; u), y) is a loss function. If time complexity is not an issue, the following optimization-based attack max DISPLAYFORM0 which is also related to the CW attack BID2, can generate strong adversarial examples, where adversarial patterns Z = (z 1, ..., z N) are unrestricted except for the bounds such as z i − x i p ≤ η. The corresponding class of adversarial patterns Z is very large, which in strong but non-generalizable adversarial examples. Non-generalizable means the perturbation z(x) has to be recomputed for every new test sample x. While the class of optimization-based attacks is powerful, its large size makes it difficult to analytically study the optimal defense methods. To make the problem learnable, we restrict the class of patterns Z to that which can be generated by a flexible but manageable class of perturbation {z(·; v) | ∀v ∈ V }, e.g., an autoencoder of a fixed architecture where the parameter v is the network weights. This class is a clearly an approximation to the class of full optimization-based attacks, but is generalizable, i.e., no time-consuming optimization is required in the test phase but only single feedforward passes. The attack network (AttNet), as we call it, can be of any class of appropriate neural networks. Here we use a three-layer fully-connected network with 300 hiddens units per layer in this paper. Different from BID21 or BID0, we feed the label y into the input of the network along with the features x. This is analogous to using the true label y in the original FGSM. While this label input is optional but it can make the training of the attacker network easier. As with other attacks, we impose the l ∞ -norm constraint on z, i.e., z(x) − x ∞ ≤ η. Suppose now f (u, v) is the empirical risk of a classifier-attacker pair where the input x is first transformed by attack network z(x; v) and then fed to the classifier g(z(x; v); u). The attack network can be trained by gradient descent as well. Given a classifier u, we can use gradient descent DISPLAYFORM1 to find an optimal attacker v that maximizes the risk f assuming the classifier u is fixed. Table 4 compares the error rates of the FGSM attacks and the attack network (AttNet). The table shows that AttNet is better than or comparable to FGSM in all cases. In particular, we already observed that the FGSM attack is no more effective against the classifier hardened against gradient-based attacks (Adv FGSM80 or Sens FGSM), but the AttNet can incur significant error (>∼ 0.9) for those hardened defenders. This indicates that the class of learning-based attacks is indeed different from the class of gradient-based attacks. Table 4: Error rates of FGSM vs learning-based attack network (AttNet) on various adversariallytrained classifiers for MNIST. FGSM-curr/AttNet-curr means they are computed/trained for the specific classifier on the leftmost column. Note that FGSM fails to attack hardened networks (Adv FGSM80 and Sens FGSM), whereas AttNet can still attack them successfully. Finally, we consider the dynamics of the pair of classifier-attacker when each player can change its parameters. Given the current classifier u, an optimal whitebox attacker parameter v is the maximizer of the risk DISPLAYFORM0 Consequently, the defender should choose the classifier parameters u such that the maximum risk is minimized u * arg min DISPLAYFORM1 This solution to the continuous minimax problem has a natural interpretation as the best worst-case solution. Assuming the attacker is optimal, i.e., it chooses the best attack from given u, no other defense can achieve a lower risk than the minimax defense u * in. The minimax defense is also a conservative defense. If the attacker is not optimal, and/or if the attack does not know the defense u exactly (as in blackbox attacks), the actual risk can be lower than what the minimax solution f (u *, v * (u *)) predicts. Before proceeding further, we point out that the claims above apply to the global minimizer u * and the maximizer function v * (·), but in practice we can only find local solutions for complex risk functions of deep classifiers and attackers. To solve FORMULA0, we analyze the problem similarly to- FORMULA7 from the previous section. At each iteration, the defender should choose u in expectation of the attack and minimize f (u, v * (u)). We use gradient descent DISPLAYFORM2 where the total derivative DISPLAYFORM3 Since the exact maximizer v * (u) is difficult to find, we only update v incrementally by one (or more) steps of gradient-ascent update DISPLAYFORM4 The ing formulation is closely related to the unrolled optimization BID17 proposed for training GANs, although the latter has a very different cost function f. Using the single update, the total derivative is DISPLAYFORM5 Similar to hardening a classifier against gradient-based attacks by minimizing FORMULA7 at each iteration, the gradient update of u for f (u, v) can be done using the gradient of the following sensitivitypenalized function DISPLAYFORM6 In other words, u is chosen not only to minimize the risk but also to prevent the attacker from exploiting the sensitivity of f to v. The algorithm is summarized in Alg. 1. Note that this algorithm is actually independent of the adversarial example problem, and can be used for other minimax problems as well. In analogy with the minimax problem, we can also consider the maximin solution defined by DISPLAYFORM0 where DISPLAYFORM1 is the minimizer function. Here we are abusing the notations for the minimax solution u *, the maximin solution v *, the minimizer u * (·), and the maximizer v * (·). Similar to the minimax solution, the maximin solution has an intuitive meaning -it is the best worst-case solution for the attacker. Assuming the defender is optimal, i.e., it chooses the best defense from that minimizes the risk f (u, v) given the attack v, no other attack can inflict a higher risk than the maximin attack v *. It is also a conservative attack. If the defender is not optimal, and/or if the defender does not know the attack v exactly, the actual risk can be higher than what the solution f (u * (v *), v * ) predicts. Note that the maximin scenario where the defender knows the attack method is not very realistic but is the opposite of the minimax scenario and provides the lower bound. To summarize, minimax and maximin defenses and attacks have the following inherent properties. Lemma 1. Let u *, v * (u), v *, u * (v) be the solutions of FORMULA0, FORMULA0, FORMULA0, FORMULA0. DISPLAYFORM2 For any given defense u, the max attack v * (u) is the most effective attack. DISPLAYFORM3 Against the optimal attack v * (u), the minimax defense u * is the most effective defense. DISPLAYFORM4 For any given attack v, the min defense u * (v) is the most effective defense. DISPLAYFORM5 Against the optimal defense u * (v), the maximin attack v * is the most effective attack. DISPLAYFORM6 The risk of the best worst-case attack is lower than that of the best worst-case defense. These properties follow directly from the definitions. The lemma helps us to better understand the dependence of defense and attack, and gives us the range of the possible risk values which can be measured empirically. To find maximin solutions, we use the same algorithm (Alg. 1) except that the variables u and v are switched and the sign of f is flipped before the algorithm is called. In addition to minimax and maximin optimization, we also consider as a reference algorithm the alternating descent/ascent method used in GAN training BID6 DISPLAYFORM0 Note that alternating descent/ascent finds local saddle points which are not necessarily minimax or maximin solutions, and therefore its solution will in general be different from the solution from Alg. 1. The difference of the solutions from three optimizations -Minimax, Maximin, and Alternating descent/ascent (Alt) -applied to a common problem, is demonstrated in FIG2. The figure shows the test error over the course of optimization starting from random initializations. One can see that Minimax (top blue curves) and Alt (middle green curves) converge to different values suggesting the learned classifiers will also be different. Table 5 compares the robustness of the classifiers trained by Minimax and Alt against the AttNet attack (1st/2nd rows and 2nd column for each η.) Minimax defense is more robust than Alt defense at η = 0.3 (0.020 vs 0.104) and at η = 0.4 (0.552 vs 0.873). For larger η's, both are unusably vulnerable. Different performance of the two classifiers implies that the minimax solution found by Alg. 1 is different from the local saddle point found by alternating descent/ascent. In addition, against FGSM attacks, Minimax is moderately robust (0.218 -0.342) despite that the classifiers are not specifically trained against gradient-based attacks. In contrast, Sens FGSM is very vulnerable (0.902 -1.000) against AttNet which we have already observed. This suggests that the class of AttNet attacks and the class of gradient-based attacks are indeed different, and the former class is larger than the latter. Table 5: Error rates of Minimax-, Alt-, and adversarially-trained (Sens FGSM) classifiers for MNIST. Minimax is overall better than Alt against AttNet-curr, and is also moderately robust against the out-of-class attack (FGSM-curr).Lastly, the adversarial examples generated by various attacks in the paper have diverse patterns and are shown in FIG4 of the appendix. We discuss some limitations of the framework and also propose an extension. Ideally, a defender should find a robust classifier against the worst attack from a very large class of attacks such as optimization-based attacks. However, it is difficult to train classifiers against attacks from a large class. On the other hand, if the class is too small, then the worst attack from that class is not representative of all possible worst attacks, and therefore the minimax defense found will not be robust to out-of-class attacks. The trade-off seems inevitable. It is, however, possible to build a defense against multiple specific types of attacks. Suppose z 1 (u),..., z m (u) are m different types of attacks, e.g., z 1 =FGSM, z 2 =IFGSM, etc. The minimax defense for the combined attack is the solution to the mixed continuous-discrete problem DISPLAYFORM0 Additionally, suppose z m+1 (u, v),..., z m+n (u, v) are n different types of learning-based attacks, e.g., z m+1 =2-layer dense net, z m+2 =5-layer convolutional nets, etc. The minimax defense against the mixture of multiple fixed-type and learning-based attacks can be found by solving DISPLAYFORM1 Due to the huge computational demand to solve, we leave it as a future work. Lastly, we discuss a bigger picture of the game between adversarial players. The minimax optimization arises in the leader-follower game BID1 with the constant sum constraint. The leader-follower setting makes sense because the defense (=classifier parameters) is often public knowledge and the attacker exploits the knowledge. Interestingly, the problem of the attack on privacy BID9 has a very similar formulation as the adversarial attack problem, different only in that the classifier is an attacker and the data perturbator is a defender. In the problem of privacy preservation against inference, the defender is a data transformer z(x) (parameterized by u) which perturbs the raw data, and the attacker is a classifier (parameterized by v) who tries to extract sensitive information such as identity from the perturbed data such as online activity of a person. The transformer is the leader, such as when the privacy mechanism is public knowledge, and the classifier is the follower as it attacks the given perturbed data. The risk for the defender is therefore the accuracy of the inference of sensitive information measured by −E[l(z(x; u), y; v)]. Solving the minimax risk problem (min u max v −E[l(z(x; u), y; v)]) gives us the best worst-case defense when the classifier/attacker knows the transformer/defender parameters, which therefore gives us a robust data transformer to preserve the privacy against the best inference attack (among the given class of attacks.) On the other hand, solving the maximin risk problem (max v min u −E[l(z(x; u), y; v)]) gives us the best worst-case classifier/attacker when its parameters are known to the transformer. As one can see, the problems of adversarial attack and privacy attack are two sides of the same coin which can be addressed by similar frameworks and optimization algorithms. In this paper, we present a continuous game formulation of adversarial attacks and defenses using a learning-based attack class implemented by neural networks. We show that this class of attacks is quite different from the gradient-based attacks. While a classifier robust to all types of attack may yet be an elusive goal, the minimax defense against the neural network-based attack class is well-defined and practically achievable. We show that the proposed optimization method can find minimax defenses which are more robust than adversarially-trained classifiers and the classifiers from simple alternating descent/ascent. We demonstrate these with MNIST and CIFAR-10. The architecture of the MNIST classifier is similar to the Tensorflow model 2, and is trained with the following hyperparameters: {Batch size = 128, optimizer = AdamOptimizer with λ = 10 −4, total # of iterations=50,000.}The attack network has three hidden fully-connected layers of 300 units, trained with the following hyperparameters: {Batch size = 128, dropout rate = 0.5, optimizer = AdamOptimizer with 10 −3, total # of iterations=30,000.} For minimax, alt, and maximin optimization, the total number of iteration was 100,000. The sensitivity-penalty coefficient of γ = 1 was used in Alg. 1. We preprocess the CIFAR-10 dataset by removing the mean and normalizing the pixel values with the standard deviation of all pixels in the image. It is followed by clipping the values to ±2 standard deviations and rescaling to [−1, 1]. The architecture of the CIFAR classifier is similar to the Tensorflow model 3 but is simplified further by removing the local response normalization layers. With the simple structure, we attained ∼ 78% accuracy with the test data. The classifier is trained with the following hyperparameters: {Batch size = 128, optimizer = AdamOptimizer with λ = 10 −4, total # of iterations=100,000.}The attack network has three hidden fully-connected layers of 300 units, trained with the following hyperparameters: {Batch size = 128, dropout rate = 0.5, optimizer = AdamOptimizer with σ = 10 −3, total # of iterations=30,000.} For minimax, alt, and maximin optimization, the total number of iteration was 100,000. The sensitivity-penalty coefficient of γ = 1 was used in Alg. 1.In the rest of the appendix, we repeat all the experiments with the MNIST dataset using the CIFAR-10 dataset. Table 8: Error rates of different attacks on various adversarially-trained classifiers for CIFAR-10. FGSM-curr means the FGSM attack on the specific classifier on the leftmost column. Adv FGSM is the classifier adversally trained with FGSM attacks. Sens FGSM is the of minimizing the sensitivity penalty. LWA FGSM is the of minimizing FORMULA7 Table 9: Error rates of FGSM vs learning-based attack network (AttNet) on various adversariallytrained classifiers for CIFAR-10. FGSM-curr/AttNet-curr means they are computed/trained for the specific classifier on the leftmost column. Note that FGSM fails to attack against the'hardened' networks (Adv FGSM80 and Sens FGSM), but AttNet can still attack them successfully.

A game-theoretic solution to adversarial attacks and defenses.

represent the short summary for finding the corresponding scientific article (long).

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.