Question: Search space is a key consideration for neural architecture search. Recently, Xie et al. (2019a) found that randomly generated networks from the same distribution perform similarly, which suggest we should search for random graph distributions instead of graphs. We propose graphon as a new search space. A graphon is the limit of Cauchy sequence of graphs and a scale-free probabilistic distribution, from which graphs of different number of vertices can be drawn. This property enables us to perform NAS using fast, low-capacity models and scale the found models up when necessary. We develop an algorithm for NAS in the space of graphons and empirically demonstrate that it can find stage-wise graphs that outperform DenseNet and other baselines on ImageNet. Neural architecture search (NAS) aims to automate the discovery of neural architectures with high performance and low cost. Of primary concern to NAS is the design of the search space, which needs to balance multiple considerations. For instance, too small a space would exclude many good solutions, whereas a space that is too large would be prohibitively expensive to search through. An ideal space should have a one-to-one mapping to solutions and sufficiently smooth in order to accelerate the search. A common technique to keep the search space manageable is to search for a small cell structure, typically containing about 10 operations with 1-2 input sources each. When needed, identical cells are stacked to form a large network. This technique allows cells found on, for instance, CIFAR-10 to work on ImageNet. Though this practice is effective, it cannot be used to optimize the overall network structure. In both manual and automatic network design, the overall network structure is commonly divided into several stages, where one stage operates on one spatial resolution and contains several nearidentical layers or multi-layer structures (i.e., cells). For example, ResNet-34 contains 4 stages with 6, 8, 12, and 6 convolutional layers, respectively. DenseNet-121 contains 4 stages with 6, 12, 24, and 16 two-layer cells. AmoebaNet-A has 3 stages, within each 6 cells are arranged sequentially. Among cells in the same stage, most connections are sequential with skip connections occasionally used. As an exception, DenseNet introduces connections between every pairs of cells within the same stage. Here we emphasize the difference between a stage and a cell. A cell typically contains about 10 operations, each taking input from 1-2 other operations. In comparison, a stage can contain 60 or more operations organized in repeated patterns and the connections can be arbitrary. A network usually contains only 3-4 stages but many more cells. In this paper, we focus on the network organization at the level of stage rather than cell. recently showed that the stage structure can be sampled from probabilistic distributions of graphs, including Erdős-Rényi (ER), Watts-Strogatz (WS), and Barabási-Albert (BA), yielding high-performing networks with low in-group variance. This finding suggests the random graph distribution, rather than the exact graph, is the main causal factor behind network performance. Thus, searching for the graph is likely not as efficient as searching for the random (c) m0 = m = 100, n = 1000 Figure 1: Three adjacency matrices of graphs generated by the Barabási-Albert model with m = m 0 = 0.1n. A black dot at location (i, j) denotes an edge from node i to node j. The sequence of matrices converges to its limit, the graphon, as n → ∞. Figure 2: Graphons for common random graph models. Different shades denote different probabilities (e.g., p and 1 − p). The Erdős-Rényi model has two parameters: number of nodes n and probability p. The Watts-Strogatz (WS) model has three parameters: number of nodes n, replacement probability p, and initial neighborhood width k. Technically, the WS model has a constant number of edges, violating exchangeability for random graphs; graphs sampled from (b) converges in probability to the same number of edges as n increases. graph distribution. The parameter space of random graph distributions may appear to be a good search space. We propose a different search space, the space of graphons, and argue for its superiority as an NAS search space. Formally introduced in Section 3, a graphon is a measurable function defined on 2 → and a probabilistic distribution from which graphs can be drawn. Graphons are limit objects of Cauchy sequences of finite graphs under the cut distance metric. Figure 1 visualizes three adjacency matrices randomly generated by the Barabási-Albert (BA) model with increasing numbers of nodes. It is easy to see that, as the number of nodes increases, the sequence of random graphs converges to its limit, a graphon. The BA model starts with an initial seed graph with m 0 nodes and arbitrary interconnections. Here we choose a complete graph as the seed. It sequentially adds new nodes until there are n nodes in the graph. For every new node v new, m edges are added, with the probability of adding an edge between v new and the node v i being proportional to the degree of v i. In Figure 1, we let m = m 0 = 0.1n. The fact that different parameterization in the same adjacency matrix suggests that directly searching in the parameter space will revisit the same configuration and is less efficient than searching in the graphon space. Additionally, graphon provides a unified and more expressive space than common random graph models. Figure 2 illustrates the graphons for the WS and the ER models. We can observe that these random models only capture a small proportion of all possible graphons. The graphon space allows new possibilities such as interpolation or striped combination of different random graph models. Finally, graphon is scale-free, so we should be able to sample an arbitrary-sized stage-wise architecture with identical layers (or cells) from a graphon. This allows us to perform expensive NAS on small datasets (e.g., CIFAR-10) using low-capacity models and obtain large stage-wise graphs to build large models. By relating graphon theory to NAS, we provide theoretically motivated techniques that scale up stage-wise graphs, which are shown to be effective in practice. Our experiments aim to fairly compare the stage-wise graphs found by our method against DenseNet and the WS random graph model by keeping other network structures and other hyperparameters constant. The indicate that the graphs found outperform the baselines consistently across a range of model capacities. The contribution of this paper revolves around building a solid connection between theory and practice. More specifically, • We propose graphon, a generalization of random graphs, as a search space for stage-wise neural architecture that consists of connections among mostly identical units. • We develop an operationalization of the theory on graphon in the representation, scaling and search of neural stage-wise graphs that perform well in fair comparisons. We review the NAS literature with a focus on the distinction between cell and stage structures. In the pioneering work of, a recurrent neural network served as the controller that outputs all network parameters for all layers without such distinction. Later works attempted to reduce the cost of search by constraining the search to cell structures only. searched for a single cell structure and perform downsampling using pooling operations.,, and searched for two types of cells: a reduction cell that includes downsampling, and a normal cell that does not. grew the cell structure from the simplest 1-operation cell to the maximum of 5 operations. DARTS relaxed discrete choices and enabled gradient-based optimization. imposed a probabilistic formulation on DARTS. While various manual designs of stage structures have been proposed (e.g., Huang et al. 12, Larsson et al. 16), NAS for stage-wise graph has received relatively little attention. found that random graphs generated by properly parameterized Watts-Strogatz models outperform manual designs for stage-wise structures. redefined the stage-wise graphs so that a node can take multiple inputs but output only a single channel, ing in large stage-wise graphs with up to 2560 nodes. evolved the connections between multiple residual blocks for applications in video processing. As a differennt type of global structure, optimized organization of downsampling and upsampling. In summary, we believe that NAS for stage structures is an emerging research direction with many unexplored opportunities. Another approach for accelerate search is to share weights among different architectures. built a lattice where a chain-structured network is a path from the beginning to the end. In ENAS, a controller is trained using gradient policy to select a subgraph, which is subsequently trained using cross-entropy. The process repeats, sharing network weights across training sessions and subgraphs. In the one-shot approach, the hypergraph is only trained once. After that, subgraphs are sampled from the hypergraph and evaluated. Finally, the best performing subgraph is retrained. Our focus in this paper is to validate that the mathematical theory of graphon can be effectively operationalized and leave weight sharing as future work. 3 ON GRAPHON The definition of graphon is straightforward, but its relation with graphs requires some standard concepts from real analysis, which we introduce below. In machine learning, graphon has found applications in hierarchical clustering and graph classification. A metric space is a space with a distance function. More specifically, a metric space is a set M with a metric function d: M × M → R that defines the distance between any two points in M and satisfies the following properties: Identity: As an example, the set of real numbers R with the absolute difference metric d abs (x, y) = |x − y| is a metric space. A Cauchy sequence is defined as a sequence (x 1, x 2, . . .) whose elements become infinitely close to each other as we move along the sequence. Formally, for any positive ∈ R +, there exists a positive integer A complete metric space is a metric space (M, d) in which every Cauchy sequence converges to a point in M. It turns out that some familiar spaces, such as rational numbers Q under the metric d abs, are not complete. As an example, the sequence defined by x 0 = 1, x n+1 = x n /2 + 1/x n converges to an irrational number √ 2. The space of real numbers R with d abs, however, is a complete metric space. Given any metric space, we can form its completion, in general, by adding limit points. In the case of graphs, the limits allow explicit descriptions. We consider a weighted undirected graph G = (V, E) where every v i ∈ V is associated with a node weight α i and every edge e ij ∈ E is associated with edge weight β ij. When necessary, we also write α i = α i (G) to highlight the graph the node belongs to. The weighted graph is a generalization of the simple graph, where all nodes have weight 1 and all edge have weight 1. Edges that do not exist are considered to have weight 0. show the following. Theorem 1. Every Cauchy sequence of weighted graphs in the metric δ converges to a graphon. Theorem 2. An weighted graphon is the limit of some Cauchy sequence in the metric δ. The definition of the cut distance δ relies on its discrete versions d andδ. To avoid unnecessary technical details, here we introduce the intuition of d and leaveδ and δ to Appendix C. For a partition S, T of V, the cut size is defined as cut(S, T, G) = vi∈S,vj ∈T α i α j β ij. When two graphs G and G = (V, E) have the same set of nodes (i.e. V = V), the partition S, T applies to both graphs. We can then define The seemingly counter-intuitive d is a sensible metric for random graphs. Consider two random graphs with n nodes from the same Erdős-Rényi model with edge density 1/2. As they are identically distributed, their distance should be small. However, their edit distance, or the number of edges where the two graphs differ, is likely quite large. In contrast, d is only O(1/n) with high probability, which is consistent with our intuition. For directed graphs with the potential of self-loops, define a digraphon as a 5-tuple (W 00, W 01, W 10, W 11, w). For nodes u i and u j, W 00 (i, j) describes the probability that no edge exists between them; W 01 (i, j) the probability that an edge goes from u i to u j; W 10 (i, j) the probability that an edge goes from u j to u i, and W 11 (i, j) the probability that two edges go both ways. w i is the probability for a self-loop at u i. For all i, j, W 00 (i, j)+W 01 (i, j)+W 10 (i, j)+W 11 (i, j) = 1. Since the graphon is a limit object, we must approaximate it with finite means. Here we use a step function approximation. We utilize a matrix B ∈ n×n as the adjacency matrix of a weighted graph whose edgeweights β ij represent the mean of a n. This approximation converges to the graphon when n tends to infinity (Lemma 3.2, Borgs et al. 3). In order to represent directed acyclic graphs in neural networks, we require the adjacency matrices to be upper-triangular and has a zero vector on the diagonal. This is equivalent to imposing a total ordering ≺ on the nodes and requiring i ≺ j for a directed edge to go from v i to v j. In the following, we propose theoretically motivated techniques for sampling stage-wise graphs of different sizes from the graphon representation and NAS for graphon. Given a finite graph G est with edge weights β ij ∈ and uniform nodeweights, we can sample a simple graph G sample (i.e. with all edgeweights equal to 0 or 1) with n nodes by drawing every edge independently as a 0-1 variable from the Bernoulli distribution parameterized by β ij. show G sample converges to G est in probability as the number of nodes n increases. This procedure requires G est and G sample to have the same number of nodes. By utilizing properties of the graphon metric space, we can sample a graph with more than n nodes from the graph G est with n nodes. When the upsampling factor is an integer k, we first create a new weighted graph G est [k] with kn nodes using the so-called k-fold blow-up procedure, and then use the above procedure to sample G sample with kn nodes. The k-fold blow-up of a graph G est is created by splitting every node in G est into k > To handle the case when the upsampling factor is not an integer, we propose a method called fractional blow-up. Suppose we want to create a stage-wise graph with kn + m nodes. We first perform k-fold blow-up to create a new graph with kn nodes and equal nodeweight 1/kn. After that, we shift the nodeweights such that the first m nodes have nodeweights 2/(kn + m) and the rest kn − m nodes have nodeweights 1/(kn + m). We subsequently split the first m node into 2 nodes, yielding a graph of equal nodeweights. As detailed in Appendix D, the cut distance between G est and G est can be bounded by where β ∆ denotes the maximum difference between any two edge weights. Since β ij ∈, β ∆ ≤ 1. From G est, we can then sample a simple graph with kn + m nodes. It is worth noting that the proposed upscaling methods differ from conventional upsampling techniques like linear or bilinear interpolation. In Appendix D.3, we show that, under moderate conditions, the k-fold blow-up graph is strictly closer to the original graph than a graph created by interpolation. We now introduce the search algorithm. When optimizing discrete objects with gradient descent, the Gumbel softmax has been widely used. Given a multinomial distribution with probabilities π 0,..., π K, we independently draw a K-dimensional vector β from a Gumbel distribution: ∀k, γ k ∼ Gumbel. The Gumbel softmax is defined as The Gumbel softmax can be understood as pitting the choices from 1 to K against each other, while the random perturbation from β enables exploration. Our search algorithm optimizes the input connections to each node in the stage-wise graph. Empirically, we find it important to let different cells in the same stage learn to collaborate during the a node on the 0-1 lattice a state on the lattice visited by SGD the estimated graphon Figure 3: The intuition on the search for the optimal graphon. When searching in the space of adjacency matrices, we only can only move on the 0-1 lattice. The optimal graphon, denoted by the filled black dot, is estimated by taking the average of the states visited by SGD. search. Therefore, we devise an algorithm that pits different input combinations against each other. Specifically, the node v i may take input from nodes v 1,..., v i−1, from which we sample K subsets. For example, for the seventh node v 7, we could sample {v 1, v 3, v 5} or {v 5, v 6} and so on. We assign a structural parameter π k to every input subset. Finding the adjacency matrix amounts to finding the best input subset for every node. We also assign separate model parameters to the same node in different input subsets. This allows all nodes in the same subset to learn to collaborate, which would be difficult if the parameters were shared across input subsets. The outputs of nodes in the same subset are aggregated using concatenation with zero padding for aligning the dimensions. During the search, the input to node v i is computed as the convex combination of the outputs from all K subsets, For the pseudo-code of the search algorithm, the reader is referred to Appendix B. We use cross-entropy loss and standard stochastic gradient descent (SGD) with momentum to optimize the model parameters together with subset weights π. After the search completes, We pick the input subset with the highest π k in the final adjacency matrix. However, the goal of the search is to find a probabilistic distribution of random graphs, not a single graph. As indicated by and, the last phase of SGD may be considered as a Markov chain that draws samples from a stationary distribution. Figure 3 illustrates this intuition. Thus, we compute the graphon as the average of adjacency matrix found by the last phase of the search. DenseNet is a classical example where the stage-wise structure plays a major role in improving the network performance. Therefore, in the experiment, we focus on improving and comparing with DenseNet. In order to create fair comparisons, we focus the search on the stage-wise structure and strictly follow DenseNet for the global network layout and the cell structure (See details in Appendix A.1). Following, we start the search from the graphon that corresponds to the WS distribution with k/n = 0.4 and p = 0.75, which we denote as WS-G(0.4, 0.75). We limit the search to input subsets that differ by 1 edge from the starting point. We perform the search on CIFAR-10 for 300 epochs with 8 cells in every stage. The search completes within 24 hours on 1 GPU. We create four groups of comparison for the four DenseNet variants: DenseNet-121, DenseNet-169, DenseNet-201, and DenseNet-264 in an increasing order of model capacity. We use the proposed scaling method (Section 4.1) to scale up the graphs in the four stages to roughly match the corresponding DenseNet variant. The largest stage-wise graph, containing 64 nodes, is used in the DenseNet-264 group. We also adjust the growth rate c so that the numbers of parameters of all models are as close to DenseNet as possible to enable fair comparisons. However, as the number of parameters depends on the stage-wise connections, which are randomly drawn from the same distribution, we do not have precise control over the number of parameters. We strictly follow the standard hyperparameters and data augmentation techniques used by DenseNet. The networks are trained for 90 epoch on ImageNet; from 6 independent training sessions are averaged. For a detailed list of hyperparameters and data augmentation, see Appendix A.2. We introduce two other baseline models besides DenseNet. In the first model, the stage-wise graphs are generated by randomly deleting edges from the fully connected graph of DenseNet. In the second model, we use random graphs generated from WS-G(0.4, 0.75), which are similar to the best random architecture in and shown to be competitive with Amoeba, PNAS and DARTS on an equal-parameter basis. For evaluation, we report the average performance on the ILSVRC-2012 validation set and the new ImageNet V2 test set. For ILSVRC-2012 validation, we report the performance directly after epoch 90. For ImageNet V2, we select the best performing model on the ILSVRC-2012 validation set from the 90 epochs and test it on ImageNet V2. In order to mitigate the effects of random variance, we sample 6 graphs from every graphon and report the average accuracy and standard deviation, as well as the number of parameters in Table 1. Across all groups of comparison and model parameter setting, the graphon found by our algorithm consistently achieves the highest accuracy despite having the least parameters. On ImageNet validation, the top-1 performance gap between our method and DensetNet is 0.4% except for 0.1% for DenseNet-121. On ImageNet V2, the performance gap is up to 0.8%. The WS-G baseline in most comparisons is stronger than DenseNet but weaker than the propose technique. We attribute the performance differences to the stage-wise graphs, since we have strictly applied the same setting, including the global network structure, the cell structure, and hyperparameter settings. The first we draw is the effectiveness of the theoretically motivated scaling technique for graphon. We scaled up the 11-node graph found by the search to graphs with up to 64 nodes in the experiments. We also scaled the WS(4, 0.25) network, initially defined for 32 nodes in, to 64 nodes in the DenseNet-264 group. The experiments show that after scaling, the relative rankings of these methods are maintained, suggesting that the proposed scaling technique incurs no performance loss. Second, we observe the standard deviations for most methods are low, even though they edge a bit higher for ImageNet V2 where model selection has been carried out. This is consistent with the findings of and reaffirms that searching for random graphs is a valid approach for NAS. Finally, we emphasize that these are created for the purpose of fair comparisons and not for showcasing the best possible performance. Our goal is to show that the graphon space and the associated cut distance metric provide a feasible approach for NAS and the empirical evidences support our argument. The design of search space is of paramount importance for neural architecture search. Recent work suggests that searching for random graph distributions is an effective strategy for the organization of layers within one stage. Inspired by mathematical theories on graph limits, we propose a new search space based on graphons, which are the limits of Cauchy sequences of graphs based on the cut distance metric. The contribution of this paper is the operationalization of the graphon theory as practical NAS solutions. First, we intuitively explain why graphon is a superior search space than the parameter space of random graph models such as the Erdős-Rényi model. Furthermore, we propose a technique for scaling up random graphs found by NAS to arbitrary size and present a theoretical analysis under the cut distance metric associated with graphon. Finally, we describe an operational algorithm that finds stage-wise graphs that outperform manually designed DenseNet as well as randomly wired architectures in. Although we find neural architectures with good performance, we remind the reader that absolute performance is not the goal of this paper. Future work involves expanding the work to different operators in the same stage graph. This can be achieved, for example, in the same manner that digraphon accommodates different types of connections. We contend that the achieved in this paper should not be considered an upper bound, but only the beginning, of what can be achieved. We believe this work opens the door toward advanced NAS algorithms in the space of graphon and the cut distance metric. A The DenseNet network contains a stem network before the first stage, which contains a 3 × 3 convolution, batch normalization, ReLU and max-pooling. This is followed by three stages for CIFAR-10 and four stages for ImageNet. Between every two stages, there is a transition block containing a 1×1 convolution for channel reduction and a 2 × 2 average pool with stride 2 for downsampling. The network ends with a 7 × 7 global average pooling and a linear layer before a softmax. Figure 4 shows the cell structure for DenseNet, which contains two convolutions with different kernel size: 1 × 1 and 3 × 3. Each of the two convolutions are immediately preceded by a batch normalization and ReLU. Every cell in the same stage outputs c channels. The input to the n th cell is the concatenation of outputs from the cell 1 to cell n − 1, for a total of c(n − 1) channels. As every cell increments the number of input channels by c, it is called the growth rate. Some detailed hyperparameters in our experiments are as follows. During the architecture search stage, we train for 300 epochs. The learning rate schedule follows DenseNet, i.e. The initial learning rate is set to 0.1, and is divided by 10 at epochs 150 and 225. Momentum is set at 0.9. Our batch size is 64 and growth rate in the DenseNet framework is set to 32. For the Gumble softmax, initial temperature is set at 1.0 and the minimum temperature is set at 0.1 with an anneal rate of 0.03. For ImageNet training, we train for 90 epochs. We use label smoothing, which assigns probability 0.1 to all labels other than the ground truth. We use a batch size of 256 = 64 per GPU × 4 GPUs or 260 = 52 per GPU × 5 GPUs, depending on the GPU memory size. For the 4 stages, growth rates are set at 26, 26, 26, 32 to match the number of parameters for Densenet-121. Learning rate is set at 0.1, divided by 10 at epochs 30, 60, and 80. We use Nesterov momentum of 0.9 and weight decay of 0.00005. We also adopt the following data augmentation for ImageNet training, which are executed sequentially: Inception-style random cropping and aspect ratio distortion, resizing the image to 224 × 224, color jitter with factors for brightness, contrast and saturation all set to 0.4, horizontal flip with probability 0.5, and AlexNet-style random changes to the brightness of the RGB channels. B THE SEARCH ALGORITHM Algorithm 1 shows the detailed procedures. Specifically, for every node v on the graph, we sample K subsets of nodes that could provide input to v, while avoiding cycles (lines 5-7). For example, for the seventh node v 7, we could sample {v 1, v 3, v 5} or {v 4, v 6} and so on. We assign one weight parameter π k to every input subset (line 8). The model parameters for node u is denoted by θ u. Every sampled input edge subset U(v, k) employ the same neural network operations in u, but with different parameters θ u,k (lines 9-11). In particular, the input to node v is the convex combination Input: totally ordered nodes V, number of subsets k 3: for each node v ∈ V do 5: I(v) ← {u | u ≺ v, u ∈ V} all nodes preceding v to avoid cycles 6: α v ← a random vector in R K weights for the subsets 9: for each input subset U(v, k) ∈ I s (v) do 10: for each node u ∈ U(v, k) do 11: INITWEIGHT(θ u,k) create a set of weights θ u,k for u Input: nodes V, input to the stage x, temperature τ, initialized Output: a feature map extracted by the current stage 15: in(v in) ← x v in is a dummy sending outputs to all nodes. 16: for each node v ∈ V do 17: for each node u ∈ U(v, k) do 19: apply the operation of u to its input 20: out(k) ←AGGREGATE(out(u, k), ∀u) sum or concatenation 21: Sample γ ∼ Gumbel 22: perform Gumbel softmax 23: v out is a dummy whose input is the stage's output. of the outputs from all K subsets. This allows all nodes in the same subset to learn to collaborate, which would be difficult if the parameters were shared across input subsets. During the forward computation, outputs from nodes in the same input subset are aggregated by either summation or concatenation (line 20). Finally, different input subsets are forced to compete by the Gumbel softmax (lines 21-23). We pick the input edge subset with the highest π as the winning adjacency matrix. For a partition S, T of V, the cut size is defined as When two graphs G and G = (V, E) have the same set of nodes (i.e. V = V), the partition S, T applies to both graphs. We can then define When the graphs have the same number of nodes, but the correspondence between nodes is unknown, the distance is defined as the minimum over all isomorphismG ∼ = G. In general, the two graphs do not have the same number of nodes. Thus, in the most general metric δ, we allow the correspondence between nodes to be fractional. We define the "overlay" matrix L ∈ R |V|×|V |. The entry L ip denotes the fractional mapping from u i ∈ V to u p ∈ V, subject to it is β pq. The cut distance δ takes the minimum over all possible overlay matrices. The fractional overlay is applicable even when the two graphs have the same node count and can lead to a lower distance than integer overlay. In fractional upsampling, we start with a graph G with n nodes and generate a new graph with n + m nodes (0 < m < n). To achieve this, we perform two operations sequentially. First, we shift the node weights such that the m nodes have weight 2/(n + m) and the rest n − m nodes have weights 1/(n + m). Next, in what we call a partial blow-up, we split each of the m nodes into two nodes. After the upsampling operation, all nodes have equal weights 1/(n + m). The requirement for equal weight is needed as the nodeweights are the probabilities of every node being sampled. Here, the k-way split of node v i is defined as replacing v i with k new nodes v i1,..., v ik, each having node weight α i /k. For any other node v j, the edge (v ik, v j) has the same weight as (v i, v j) and the same applies to (v j, v ik). In the following, we analyze the cut distance between the new graph and the original graph and show its upper bound is (n−m)m n(n+m) β ∆, where β ∆ denotes the maximum difference between any two edgeweights. Theorem 3. Let G = (V, E) and G = (V, E) be two weighted graphs that have the same set of n nodes with different nodeweights and the same set of edges with the same edgeweights. If every node in G has nodeweight 1/n, whereas G has m nodes (0 < m < n) with nodeweight 2/(n + m) and n − m nodes having weight 1/(n + m), then where β ∆ denotes the maximum difference between any two edge weights. Proof. We create an overlay matrix L with the diagonal terms L ii = min(α i (G), α i G ). For the first m diagonal terms, L ii = 1/n, and for the rest the diagonal terms are 1/(n + m). Then If i = p and j = q, then β ij (G) = β pq (G). Thus, Note that G[k] and G[itpl(k)] have equal number of nodes. By definition, for a partition S, T of the node set V, Now we construct the partition S, T such that the first km nodes belong to S and the rest kn − km nodes belong to T. Since the adjacency matrix of G is upper triangular and non-zero, without loss of generality we pick m such that m i=1 β im (G) = n i=m+1 β im (G). Since all entries below the diagonal are zero, that inequality is satisfied as long as the column sum at the index m is non-zero. Hence, we let S = {v 1 . . . v km} and T = {v km+1, . . ., v kn}. Let the cut distance under this partition S, we have proven the desired proposition. Theorem 5 shows the k-fold blow-up method is a better approximation of the original graph in terms of the cut distance δ than the 1D linear interpolation. But the exact k-fold blow-up is only applicable when k is an integer. If a graph of size n + m(0 < m < n) is desired, we need to resort to the fractional blow-up method, which has been analyzed in Theorems 3 and 4. We show that when m is 1 or n − 1, this partial blowup operation does not cause δ to change more than O(β ∆ /n). However, when m is n/2, δ between the original graph and the new graph could be up to β ∆ /6. This suggests that the fractional upsampling in a graph that is similar to the original when only a small number of nodes (relative to n) is added.

Represent the TLDR for finding the corresponding scientific article

Graphon is a good search space for neural architecture search and empirically produces good networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the scientific article to find a short TLDR

Question: Adversarial training is by far the most successful strategy for improving robustness of neural networks to adversarial attacks. Despite its success as a defense mechanism, adversarial training fails to generalize well to unperturbed test set. We hypothesize that this poor generalization is a consequence of adversarial training with uniform perturbation radius around every training sample. Samples close to decision boundary can be morphed into a different class under a small perturbation budget, and enforcing large margins around these samples produce poor decision boundaries that generalize poorly. Motivated by this hypothesis, we propose instance adaptive adversarial training -- a technique that enforces sample-specific perturbation margins around every training sample. We show that using our approach, test accuracy on unperturbed samples improve with a marginal drop in robustness. Extensive experiments on CIFAR-10, CIFAR-100 and Imagenet datasets demonstrate the effectiveness of our proposed approach. A key challenge when deploying neural networks in safety-critical applications is their poor stability to input perturbations. Extremely tiny perturbations to network inputs may be imperceptible to the human eye, and yet cause major changes to outputs. One of the most effective and widely used methods for hardening networks to small perturbations is "adversarial training" , in which a network is trained using adversarially perturbed samples with a fixed perturbation size. By doing so, adversarial training typically tries to enforce that the output of a neural network remains nearly constant within an p ball of every training input. Despite its ability to increase robustness, adversarial training suffers from poor accuracy on clean (natural) test inputs. The drop in clean accuracy can be as high as 10% on CIFAR-10, and 15% on Imagenet , making robust models undesirable in some industrial settings. The consistently poor performance of robust models on clean data has lead to the line of thought that there may be a fundamental trade-off between robustness and accuracy , and recent theoretical characterized this tradeoff (; ;). In this work, we aim to understand and optimize the tradeoff between robustness and clean accuracy. More concretely, our objective is to improve the clean accuracy of adversarial training for a chosen level of adversarial robustness. Our method is inspired by the observation that the constraints enforced by adversarial training are infeasible; for commonly used values of, it is not possible to achieve label consistency within an -ball of each input image because the balls around images of different classes overlap. This is illustrated on the left of Figure 1, which shows that the -ball around a "bird" (from the CIFAR-10 training set) contains images of class "deer" (that do not appear in the training set). If adversarial training were successful at enforcing label stability in an = 8 ball around the "bird" training image, doing so would come at the unavoidable cost of misclassifying the nearby "deer" images that come along at test time. At the same time, when training images lie far from the decision boundary (eg., the deer image on the right in Fig 1), it is possible to enforce stability with large with no compromise in clean accuracy. When adversarial training on CIFAR-10, we see that = 8 is too large for some images, causing accuracy loss, while being unnecessarily small for others, leading to sub-optimal robustness. Figure 1: Overview of instance adaptive adversarial training. Samples close to the decision boundary (bird on the left) have nearby samples from a different class (deer) within a small L p ball, making the constraints imposed by PGD-8 / PGD-16 adversarial training infeasible. Samples far from the decision boundary (deer on the right) can withstand large perturbations well beyond = 8. Our adaptive adversarial training correctly assigns the perturbation radius (shown in dotted line) so that samples within each L p ball maintain the same class. The above observation naturally motivates adversarial training with instance adaptive perturbation radii that are customized to each training image. By choosing larger robustness radii at locations where class manifolds are far apart, and smaller radii at locations where class manifolds are close together, we get high adversarial robustness where possible while minimizing the clean accuracy loss that comes from enforcing overly-stringent constraints on images that lie near class boundaries. As a , instance adaptive training significantly improves the tradeoff between accuracy and robustness, breaking through the pareto frontier achieved by standard adversarial training. Additionally, we show that the learned instance-specific perturbation radii are interpretable; samples with small radii are often ambiguous and have nearby images of another class, while images with large radii have unambiguous class labels that are difficult to manipulate. Parallel to our work, we found that uses adaptive margins in a max-margin framework for adversarial training. Their work focuses on improving the adversarial robustness, which differs from our goal of understanding and improving the robustness-accuracy tradeoff. Moreover, our algorithm for choosing adaptive margins significantly differs from that of. Adversarial attacks are data items containing small perturbations that cause misclassification in neural network classifiers . Popular methods for crafting attacks include the fast gradient sign method (FGSM) which is a one-step gradient attack, projected gradient descent (PGD) which is a multi-step extension of FGSM, the C/W attack , DeepFool (, and many more. All these methods use the gradient of the loss function with respect to inputs to construct additive perturbations with a norm-constraint. Alternative attack metrics include spatial transformer attacks , attacks based on Wasserstein distance in pixel space , etc. Defending against adversarial attacks is a crucial problem in machine learning. Many early defenses (; ;), were broken by strong attacks. Fortunately, adversarially training is one defense strategy that remains fairly resistant to most existing attacks. denote the set of training samples in the input dataset. In this paper, we focus on classification problems, hence, y i ∈ {1, 2, . . . N c}, where N c denotes the number of classes. Let f θ (x): R c×m×n → R Nc denote a neural network model parameterized by θ. Classifiers are often trained by minimizing the cross entropy loss given by whereỹ i is the one-hot vector corresponding to the label y i. In adversarial training, instead of optimizing the neural network over the clean training set, we use the adversarially perturbed training set. Mathematically, this can be written as the following min-max problem This problem is solved by an alternating stochastic method that takes minimization steps for θ, followed by maximization steps that approximately solve the inner problem using k steps of PGD. For more details, refer to. end if 8: S + = {i|f (x i) is correctly classified as y i } S − = {i|f (x i) is incorrectly classified as y i } 11: 12: end for To remedy the shortcomings of uniform perturbation radius in adversarial training (Section 1), we propose Instance Adaptive Adversarial Training (IAAT), which solves the following optimization: Like vanilla adversarial training, we solve this by sampling mini-batches of images {x i}, crafting adversarial perturbations {δ i} of size at most {i}, and then updating the network model using the perturbed images. The proposed algorithm is distinctive in that it uses a different i for each image x i. Ideally, we would choose each i to be as large as possible without finding images of a different class within the i -ball around x i. Since we have no a-priori knowledge of what this radius is, we use a simple heuristic to update i after each epoch. After crafting a perturbation for x i, we check if the perturbed image was a successful adversarial example. If PGD succeeded in finding an image with a different class label, then i is too big, so we replace i ← i − γ. If PGD failed, then we set i ← i + γ. Since the network is randomly initialized at the start of training, random predictions are made, and this causes {i} to shrink rapidly. For this reason, we begin with a warmup period of a few (usually 10 epochs for CIFAR-10/100) epochs where adversarial training is performed using uniform for every sample. After the warmup period ends, we perform instance adaptive adversarial training. A detailed training algorithm is provided in Alg. 1. Set i = 2 8: else 9: To evaluate the robustness and generalization of our models, we report the following metrics: test accuracy of unperturbed (natural) test samples, adversarial accuracy of white-box PGD attacks, adversarial accuracy of transfer attacks and accuracy of test samples under common image corruptions . Following the protocol introduced in , we do not train our models on any image corruptions. On CIFAR-10 and CIFAR-100 datasets, we perform experiments on Resnet-18 and WideRenset-32-10 models following . All models are trained on PGD-10 attacks i.e., 10 steps of PGD iterations are used for crafting adversarial attacks during training. In the whitebox setting, models are evaluated on: PGD-10 attacks with 5 random restarts, PGD-100 attacks with 5 random restarts, and PGD-1000 attacks with 2 random restarts. For transfer attacks, an independent copy of the model is trained using the same training algorithm and hyper-parameter settings, and PGD-1000 adversarial attacks with 2 random restarts are crafted on the surrogate model. For image corruptions, following , we report average classification accuracy on 19 image corruptions. Beating the robustness-accuracy tradeoff: In adversarial training, the perturbation radius is a hyper-parameter. Training models with varying produces a robustness-accuracy tradeoff curvemodels with small training achieve better natural accuracy and poor adversarial robustness, while models trained on large have improved robustness and poor natural accuracy. To generate this tradeoff, we perform adversarial training with in the range {1, 2, . . . 8}. Instance adaptive adversarial training is then compared with respect to this tradeoff curve in Fig. 3a, 3b. Two versions of IAAT are reported -with and without a warmup phase. In both versions, we clearly achieve an improvement over the accuracy-robustness tradeoff. Use of the warmup phase helps retain robustness with a drop in natural accuracy compared to its no-warmup counterpart. Clean accuracy improves for a fixed level of robustness: On CIFAR-10, as shown in Table. 1, we observe that our instance adaptive adversarial training algorithm achieves similar adversarial robustness as the adversarial training baseline. However, the accuracy on clean test samples increases by 4.06% for Resnet-18 and 4.49% for WideResnet-32-10. We also observe that the adaptive training algorithm improves robustness to unseen image corruptions. This points to an improvement in overall generalization ability of the network. On CIFAR-100 (Table. 2), the performance gain in natural test accuracy further increases -8.79% for Resnet-18, and 9.22% for Wideresnet-32-10. The adversarial robustness drop is marginal. Maintaining performance over a range of test: Next, we plot adversarial robustness over a sweep of values used to craft attacks at test time. Fig. 4a, 4b shows an adversarial training baseline with = 8 performs well at high regimes and poorly at low regimes. On the other hand, adversarial training with = 2 has a reverse effect, performing well at low and poorly at high regimes. Our instance adaptive training algorithm maintains good performance over all regimes, achieving slightly less performance than the = 2 model for small test, and dominating all models for larger test. Interpretability of: We find that the values of i chosen by our adaptive algorithm correlate well with our own human concept of class ambiguity. Figure 2 (and Figure 6 in Appendix B) shows that a sampling of images that receive small i contains many ambiguous images, and these images are perturbed into a (visually) different class using = 16. In contrast, images that receive a large i have a visually definite class, and are not substantially altered by an = 16 perturbation. Robustness to other attacks: While our instance adaptive algorithm is trained on PGD attacks, we are interested to see if the trained model improves robustness on other adversarial attacks. As shown in Table. 3, IAAT achieves similar level of robustness as adversarial training on other gradient-based attacks, while improving the natural accuracy. , we attack Imagenet models using random targeted attacks instead of untargeted attacks as done in previous experiments. During training, adversarial attacks are generated using 30 steps of PGD. As a baseline, we use adversarial training with a fixed of 16/255. This is the setting used in. Adversarial training on Imagenet is computationally intensive. To make training practical, we use distributed training with synchronized SGD on 64/128 GPUs. More implementation details can be found in Appendix E. At test time, we evaluate the models on clean test samples and on whitebox adversarial attacks with = {4, 8, 12, 16}. PGD-1000 attacks are used. Additionally, we also report normalized mean corruption error (mCE), an evaluation metric introduced in to test the robustness of neural networks to image corruptions. This metric reports mean classification error of different image corruptions averaged over varying levels of degradation. Note that while accuracies are reported for natural and adversarial robustness, mCE reports classification errors, so lower numbers are better. Our experimental are reported in Table. 4. We observe a huge drop in natural accuracy for adversarial training (25%, 22% and 20% drop for Resnet-50, 101 and 152 respectively). Adaptive adversarial training significantly improves the natural accuracy -we obtain a consistent performance gain of 10+% on all three models over the adversarial training baseline. On whitebox attacks, IAAT outperforms the adversarial training baseline on low regimes, however a drop of 13% is observed at high's (= 16). On the corruption dataset, our model consistently outperforms adversarial training. Recall from Section 3 that during warmup, adversarial training is performed with uniform normbound constraints. Once the warmup phase ends, we switch to instance adaptive training. From Table 5 and 6, we observe that when warmup is used, adversarial robustness improves with a small drop in natural accuracy, with more improvements observed in CIFAR-100. However, as shown in Fig. 3a and 3b, both these settings improve the accuracy-robustness tradeoff. We are interested in estimating instance-specific perturbation radius i such that predictions are consistent within the chosen i -ball. To obtain an exact estimate of such an i, we can perform a line search as follows: Given a discretization η and a maximum perturbation radius max, generate PGD attacks with radii {iη}. Choose the desired i as the maximum iη for which the prediction remains consistent as that of the ground-truth label. We compare the performance of exact line search with that of IAAT in Table 7. We observe that exact line search marginally improves compared to IAAT. However, exact line search is computationally expensive as it requires performing max /η additional PGD computations, whereas IAAT requires only 2. In this work, we focus on improving the robustness-accuracy tradeoff in adversarial training. We first show that realizable robustness is a sample-specific attribute: samples close to the decision boundary can only achieve robustness within a small ball, as they contain samples from a different class beyond this radius. On the other hand samples far from the decision boundary can be robust on a relatively large perturbation radius. Motivated by this observation, we develop instance adaptive adversarial training, in which label consistency constraints are imposed within sample-specific perturbation radii, which are in-turn estimated. Our proposed algorithm has empirically been shown to improve the robustness-accuracy tradeoff in CIFAR-10, CIFAR-100 and Imagenet datasets. A recent paper that addresses the problem of improving natural accuracy in adversarial training is mixup adversarial training , where adversarially trained models are optimized using mixup loss instead of the standard cross-entropy loss. In this paper, natural accuracy was shown to improve with no drop in adversarial robustness. However, the robustness experiments were not evaluated on strong attacks (experiments were reported only on PGD-20). We compare our implementation of mixup adversarial training with IAAT on stronger attacks in Table. 8. We observe that while natural accuracy improves for mixup, drop in adversarial accuracy is much higher than IAAT. A visualization of samples from CIFAR-10 dataset with the corresponding value assigned by IAAT is shown in Figure. 5. We observe that samples for which low's are assigned are visually confusing (eg., top row of Figure. 5), while samples with high distinctively belong to one class. In addition, we also show more visualizations of samples near decision boundary which contain samples from a different class within a fixed ∞ ball in Figure. 6. The infeasibility of label consistency constraints within the commonly used perturbation radius of ∞ = 8 is apparent in this visualization. Our algorithm effectively chooses an appropriate that retains label information within the chosen radius. Next, we visualize the evolution of over epochs in adaptive adversarial training. A plot showing the average growth, along with the progress of 3 randomly picked samples are shown in Fig. 7a and 7b. We observe that average converges to around 11, which is higher than the default setting of = 8 used in adversarial training. Also, each sample has a different profile -for some, increases well beyond the commonly use radius of = 8, while for others, it converges below it. In addition, a plot showing the histogram of's at different snapshots of training is shown in Fig. 8. We observe an increase in spread of the histogram as the training progresses. Testing against a strong adversary is crucial to assess the true robustness of a model. A popular practice in adversarial robustness community is to attack models using PGD with many attack iterations . So, we test our instance adaptive adversarially trained models on a sweep of PGD iterations for a fixed level. Following , we perform the sweep upto 2000 attack steps fixing = 16. The ing plot is shown in Figure. 9. For all three Resnet models, we observe a saturation in adversarial robustness beyond 500 attack iterations. As shown in Alg. 2, IAAT algorithm has two hyper-parameters -smoothing constant β and discretization γ. In this section, we perform a sensitivity analysis of natural and robust accuracies by varying these hyper-parameters. Results are reported in Table. 9. We observe that the algorithm is not too sensistive to the choice of hyper-parameters. But the best performance is obtained for γ = 1.9 and β = 0.1. On CIFAR-10 and CIFAR-100 datasets, our implementation follows the standard adversarial training setting used in. During training, adversarial examples are generated using PGD-10 attacks, which are then used to update the model. All hyperparameters we used are tabulated in Table. 10. For Imagenet implementation, we mimic the setting used in. During training, adversaries are generated with PGD-30 attacks. This is computationally expensive as every training update is followed by 30 backprop iterations to generate the adversarial attack. To make training feasible, we perform distributed training using synchronized SGD updates on 64 / 128 GPUs. We follow the training recipe introduced in for large batch training. Also, during training, adversarial attacks are generated with FP-16 precision. However, in test phase, we use FP-32. We further use two more tricks to speed-up instance adaptive adversarial training: A weaker attacker(PGD-10) is used in the algorithm for selecting (Alg. 2). After i is selected per Alg. 2, we clip it with a lower-bound i.e., i ← max(i, lb). lb = 4 was used in our experiments. Hyperparameters used in our experiments are reported in Table 11. All our models were trained on PyTorch. Resnet-50 model was trained on 64 Nvidia V100 GPUs, while Resnet-101 and Resnet-152 models were trained on 128 GPUs. Time taken for instance adaptive adversarial training for all models is reported in Table. 12.

Instance adaptive adversarial training for improving robustness-accuracy tradeoff

Represent the TLDR for finding the corresponding scientific article

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: represent the scientific article to find a short tldr

Question: Machine learning models including traditional models and neural networks can be easily fooled by adversarial examples which are generated from the natural examples with small perturbations. This poses a critical challenge to machine learning security, and impedes the wide application of machine learning in many important domains such as computer vision and malware detection. Unfortunately, even state-of-the-art defense approaches such as adversarial training and defensive distillation still suffer from major limitations and can be circumvented. From a unique angle, we propose to investigate two important research questions in this paper: Are adversarial examples distinguishable from natural examples? Are adversarial examples generated by different methods distinguishable from each other? These two questions concern the distinguishability of adversarial examples. Answering them will potentially lead to a simple yet effective approach, termed as defensive distinction in this paper under the formulation of multi-label classification, for protecting against adversarial examples. We design and perform experiments using the MNIST dataset to investigate these two questions, and obtain highly positive demonstrating the strong distinguishability of adversarial examples. We recommend that this unique defensive distinction approach should be seriously considered to complement other defense approaches. Machine learning models including SVMs BID0 and especially deep neural networks BID17 can be easily fooled by adversarial examples which are generated from the natural examples with small perturbations. Quite often, both machine learning models and humans can classify the natural examples such as the images of pandas with high accuracy, and humans can still classify the adversarial examples as pandas with high accuracy because the small perturbations are imperceptible; however, machine learning models are fooled to misclassify adversarial examples as some targets such as gibbons BID4 desired by attackers. This intriguing property or vulnerability of machine learning models poses a critical challenge to machine learning security, and it impedes the wide application of machine learning in many important domains such as computer vision (e.g., for self driving cars) and even in malware detection BID0 BID19. Furthermore, the discovery of new and powerful adversarial example generation methods such as BID17 BID4 BID1 BID2 BID7 BID9 BID11 BID10 BID16 goes on without cessation, indicating to a certain extent the unlimited capabilities for attackers to continuously and easily fool machine learning models. On the other hand, even state-of-the-art defense approaches such as adversarial training BID17 BID4 and defensive distillation BID12 still suffer from major limitations and can be circumvented (Section 2). Therefore, the unfortunate status quo is that attackers prevail over defenders. In this paper, from a unique angle, we propose to investigate two important research questions that concern the distinguishability of adversarial examples. Question 1: are adversarial examples distinguishable from natural examples? Question 2: are adversarial examples generated by different methods distinguishable from each other? If the answer to Question 1 will be positive, i.e., given a certain classification task such as image classification, generated adversarial examples (regardless of the objects they represent) largely belong to one class while natural examples belong to the other class, then defenders can simply discard those adversarial examples to protect the machine learning models. If the answer to Question 2 will be positive, i.e., adversarial examples generated by different methods clearly belong to different classes, defenders can better protect the machine learning models, for example, by incorporating the corresponding examples into the adversarial training process to enhance the robustness of the models. Besides such practical benefits, answering these two questions may also help researchers further identify the nature of adversarial examples. Formally, we consider a classification problem in adversarial environments as a multi-label classification problem. That is, upon seeing a new input such as an image, while the original task such as classifying the image as a certain object is important, it is also important to classify the image as a generated adversarial vs. a natural example in the first place. We formulate this multi-label classification problem in Section 3 to guide us in answering the two questions, and term the corresponding defense approach as defensive distinction, which distinguishes adversarial vs. natural examples and distinguishes adversarial examples generated by different methods to protect against the attacks. We design and perform experiments using the MNIST dataset to investigate the two research questions and evaluate the effectiveness of our defensive distinction approach. In our experiments, we consider multiple scenario-case combinations that defenders either know or do not know the neural network, source images, and methods as well as parameters used by attackers for generating adversarial examples. We obtain highly positive answers to both research questions. For example, in some typical cases, adversarial vs. natural examples can be distinguished perfectly with 100% accuracy, while adversarial examples generated by different methods can be distinguished with over 90% accuracy. Our experimental demonstrate the strong distinguishability of adversarial examples, and demonstrate the value of the defensive distinction approach. We recommend that this unique defense approach should be seriously considered to complement other defense approaches. We make four main contributions in this paper: we propose to investigate two important research questions that concern the distinguishability of adversarial examples; we formulate a classification problem in adversarial environments as a multi-label classification problem to answer the two questions; we propose and explore a unique defense approach termed as defensive distinction; we design and perform experiments to empirically demonstrate the strong distinguishability of adversarial examples and the value of our defensive distinction approach. Adversarial examples can easily fool machine learning models, and they can be generated by a number of powerful methods. One representative method is L-BFGS BID17, which formulates the adversarial example generation problem as a box-constrained optimization problem and uses line-search to identify approximated optimal solutions. Another representative method is FGSM (Fast Gradient Sign Method) BID4, which efficiently obtains an optimal max-norm constrained perturbation for generating adversarial examples. FGSM inspired researchers to pursue a few fine-grained refinements over it, ing in powerful methods such as BIM (Basic Iterative Method) by BID7, PGD (Projected Gradient Descent) by BID9, and MIM (Momentum Iterative Method) by. L-BFGS also inspired researchers to design more powerful or general methods such as C&W by BID1 and EAD (Elastic Net Method) by BID2. JSMA (Jacobian-based Saliency Map Approach) BID11 ) is another representative method, which constructs adversarial saliency maps to guide the selection of input features for perturbation in multiple iterations. On the defense side, one representative approach is adversarial training BID17 BID4, which basically uses both the generated adversarial examples and the natural (a.k.a clean) examples to train the machine learning models with better regularization. Another representative approach is defensive distillation BID12, which basically uses the same neural network architecture to train a distilled network based on the probability vectors produced by the original network, thus reducing the models' sensitivity to small input perturbations. Both approaches in essence focus on improving the generalization and thus robustness of the machine learning models that perform the original classification task. While representing the state-of-theart, both approaches still suffer from major limitations and can be circumvented. For example, defensive distillation was found to be ineffective against C&W attacks BID1 and black-box attacks BID13, while adversarial training was found to be ineffective against adversarial examples generated by many iterative methods such as BIM BID6, , and MIM.Overall, powerful adversarial example generation methods exist and will continue to be proposed, while existing defense approaches are far from being perfect yet. The unfortunate status quo is that attackers prevail over defenders. Our work is likely one step toward making a change. We now present the problem formulation and the corresponding defensive distinction protection approach; we also define a threat model that can be used for evaluating our approach. To date, a classification problem in adversarial environments is viewed as a single-label classification problem, where a function or model f: X → Y is learned to assign each input example x i ∈ X a single label y i ∈ Y, where the range Y is a set of m singletons representing m possible classes such as 10 possible digits. In other words, a single concept or semantic meaning is associated with each example. State-of-the-art defense approaches such as adversarial training and defensive distillation reviewed in Section 2 do not change this basic problem formulation. However, considering the two important distinguishability questions that we propose to investigate (Question 1: are adversarial examples distinguishable from natural examples? Question 2: are adversarial examples generated by different methods distinguishable from each other?), it is straightforward for us to formulate a classification problem in adversarial environments as a multi-label classification problem. That is, a function or model f: X → Y is learned to assign each input example x i a pair of labels represented by DISPLAYFORM0 Here, the new range Y = Y × Z is the Cartesian product of Y (i.e., the range in the original classification task) and Z, where Z is a set of n singletons representing n possible classes regarding if x i is an adversarial example. In the simplest situation, n = 2 and b i indicates either an example is natural (i.e., clean) or adversarial. In a more complex situation, n > 2 and b i can indicate either an example is natural or generated by one of n − 1 different adversarial example generation methods. Under this new problem formulation, two concepts or semantic meanings 1 are associated with each example. The solution to the problem will help us answer the two research questions, and help defenders simply discard or leverage the identified adversarial examples to better protect their machine learning models as described in Section 1. Moreover, one major advantage of a multi-label classification formulation and the corresponding multi-label learning is on exploring the intrinsic correlations between multiple labels. In real-world adversarial environments, this advantage is significant because it will allow researchers and defenders to perform in-depth analysis of the behavior of attackers. For example, do attackers create most adversarial examples targeting at some specific classes such as gibbons? do attackers use different generation methods for different targeted classes? and, do attackers use examples of some specific source classes to create adversarial examples targeting at some specific classes (Appendix A)? These and perhaps other types of label correlation analyses will be valuable for inferring the real intentions of attackers, and even for revolutionizing the design of the original machine learning systems. We simply define the defensive distinction protection approach as training a model for a multi-label classification problem formulated in adversarial environments to identify adversarial examples and protect against them. So a training set will include both natural examples and generated adversarial examples with their ground truth b i values correspondingly labeled in y i = (a i, b i) ∈ Y. The ground truth a i values for natural examples are obviously known based on some existing dataset such as MNIST. The ground truth a i values for adversarial examples can be missing BID20 or not needed (as in our solution described below) depending on specific learning algorithms. Many multi-label learning algorithms have been proposed with different strategies or considerations on the order-of-correlations among labels BID21. In this paper, we experiment with the simplest first-order strategy that basically transforms a multi-label classification problem into multiple single-label problems. More specifically, three models will be independently trained:• DDP-Model (defensive distinction primary model): a single-label binary model for distinguishing an input example x i as either natural or adversarial.• Original-Model: a single-label multi-class model for the original task on classifying the input example x i as one of multiple possible objects such as one of 10 digits.• DDS-Model (defensive distinction secondary model): a single-label multi-class model 2 for distinguishing known adversarial examples as generated by different specific methods. This simple strategy provides a baseline solution to the defensive distinction approach. It is simple in model formulation, computation, and evaluation, albeit high-order strategies would be more powerful in exploring label correlations BID15 BID21. In existing defense approaches such as adversarial training and defensive distillation, a threat model only needs to define the capabilities of attackers because those approaches focus on improving the generalization and thus robustness of the machine learning models that perform the original classification task. Typically, two types of attacks are defined: white-box attacks that have the full knowledge about the original machine learning model including its architecture, parameter values, and training data, and thus can replicate the exact model to generate adversarial examples, and black-box attacks that only have the limited knowledge about the original machine learning model such as the labels for a given set of input examples, and thus often need to create another "substitute model" to generate adversarial examples BID11.In our defensive distinction approach, a thread model needs to further consider the capabilities of defenders because the effectiveness of our approach is related to the defenders' knowledge about the machine learning model (including its architecture, parameter values, and training data), adversarial example generation methods, parameters (e.g., maximum distortion) of the generation methods, and source examples used by attackers. We refer to these factors as AdvGen-Model, AdvGen-Methods, AdvGen-Parameters, and Adv-Examples. These considerations together with the capabilities of attackers appear to complicate the overall threat model; however, starting directly from the perspective of defenders, we can still clearly define some representative scenarios and cases as shown in TAB0. Briefly, Scenarios 1 and 2 represent that an attacker's AdvGen-Model is known or unknown to a defender, respectively; Cases 1 to 4 represent the four combinations of the defender's knowledge on AdvGen-Parameters and Adv-Examples, respectively. In total, eight scenario-case combinations (from S1C1 to S2C4) or simply cases are considered in this paper. Note that we do not include AdvGen-Methods in the table because it is reasonable to assume that popular adversarial example generation methods are known to both defenders and attackers (similar to the basic assumption in cryptographic systems that algorithms should not be considered as secrets and could be known by both defenders and attackers); however, we do consider the AdvGen-Methods factor in some of our experiments in the next section by leaving out certain generation method in model training. We now evaluate the effectiveness of our defensive distinction approach by performing three sets of experiments with two for DDP-Model and one for DDS-Model. We design our experiments by using the MNIST dataset of handwritten digits and two different convolutional neural networks (CNNs). The first CNN is a slight variation of the original LeNet-5 BID8 by adding rectified linear units to the convolutional layers and using max pooling in the sub-sampling layers. We refer to this network as LeNet-5. The second CNN network has a convolutional layer with 64 filters followed by two convolutional layers with 128 filters, one 2 by 2 max pooling layer, one fully connected layer with 64 rectified linear units, and one softmax layer. We refer to this network as Basic-CNN. When we train these two CNNs for different classification tasks in our experiments, we use learning rate η=0.001, number of epochs=6, and batch size=128. For the original 10-digit classification task, LeNet-5 and Basic-CNN achieve 99.05% and 98.99% accuracy, respectively, on the 10,000 images from the MNIST test set. In the generation of adversarial examples, one LeNet-5 network trained with the 60,000 images from the MNIST training set is considered as the attacker's model (AdvGen-Model) and six generation methods including FGSM, JSMA, BIM, L-BFGS, MIM, and PGD (Section 2) are used by leveraging the v2.1.0 of the CleverHans library. Based on each source example (representing one digit) chosen from the MNIST test set, nine target adversarial example generation attempts (for the rest nine digit classes) are made. In other words, we consider targeted attacks, which are more damaging than untargeted attacks that simply misclassify source examples. We generate five adversarial example datasets using the six methods based on different method parameters and source examples. As shown in TAB1, datasets adv tr and adv C1 are generated by using the same source examples with the index range 0-299 for each digit class such as'0', i.e., the first 300 examples from each digit class in the MNIST test set are chosen as source examples. These two datasets also share the same method parameters, but they are different due to the difference in the initial randomization of their generation. Dataset adv C2 has the same method parameters but different source examples compared with adv tr. Dataset adv C3 has the same source examples but different method parameters compared with adv tr. Dataset adv C4 has different source examples and different parameters compared with adv tr. The total number of adversarial examples in each dataset is approximately 300×9×6=16,200. Adversarial examples in adv tr will be used for training our DDP-Model and DDS-Model defensive distinction models based on LeNet-5 and Basic-CNN. Models trained based on LeNet-5 represent the Scenario 1 in TAB0 because the AdvGen-Model also uses LeNet-5, while models trained based on Basic-CNN represent the Scenario 2 in TAB0. Adversarial examples in adv C1 to adv C4 will be used for testing our defensive distinction models, corresponding to the Cases C1 to C4 in TAB0. So now, our design and setup of experiments allow us to evaluate our defensive distinction approach for all the eight scenario-case combinations from S1C1 to S2C4 TAB0. The simplest DDP-Model is a binary classifier for distinguishing natural examples from adversarial examples generated by a single attack method. For each method, a classifier is trained on a mixture of 1,000 natural examples from the MNIST training set and 1,000 adversarial examples from the dataset adv tr in TAB1 corresponding to the selected method. The classifier is then evaluated on a mixture of 1,000 natural examples from the MNIST test set and 1,000 adversarial examples from one test dataset adv C1 to adv C4 in TAB1 corresponding to the selected method. Note that in this paper, examples are all randomly selected from the corresponding sets; each experiment including the training and testing is independently performed for 10 runs to present the average . Model accuracy for the four test cases with known AdvGen-Parameters (S1C1, S1C2, S2C1, S2C2) are given in FIG1. Classification accuracy is very high for all the six methods, suggesting that regardless of the defender's knowledge on AdvGen-Model (S1 vs. S2) and AdvGen-Examples (C1 vs. C2), a simple DDP-Model can be highly effective as long as AdvGen-Parameters are known. Across the four test cases, the models for distinguishing JSMA examples from natural examples struggled the most but still achieved high average accuracy at 92.35%, while the models for other methods such as MIM, PGD, FGSM, and BIM classified examples perfectly. One reason for the imperfect on the JSMA examples could be that JSMA only selects a small number such as a pair of input features (i.e., pixels) for perturbation in each iteration BID11. Model accuracy for the remaining four test cases with unknown AdvGen-Parameters (S1C3, S1C4, S2C3, S2C4) are given in FIG1. Across the four test cases, classification accuracy is still very high (> 95%) for L-BFGS, FGSM, and BIM, suggesting that a simple DDP-Model can still be highly effective for these three methods even when some AdvGen-Parameters are unknown to a defender. Models for the other three methods did not perform well. Notably, the models only achieved near 50% accuracy (i.e., like random guessing) for MIM, PGD, and JSMA examples under the test cases S1C4 and S2C4, and only performed slightly better than random guessing for MIM and PGD examples under the test cases S1C3 and S2C3; these suggest that knowing the parameters of some adversarial example generation methods is important to defenders. DISPLAYFORM0 Overall, the in FIG1 are highly informative in practice for defenders. Most importantly, it is always beneficial for defenders to train a DDP-Model by using adversarial examples generated based on a variety of parameters. Meanwhile, the exact model architecture and the exact natural examples used by attackers are not influential in the accuracy of the defenders' models. So far, we analyzed the accuracy of DDP-Model without considering whether an adversarial example successfully fooled the Original-Model for the 10-digit classification task to misclassify it as a targeted digit class. Back to our multi-label classification formulation of the problem in Section 3.1, correlating the labels of these two models on an adversarial example will be interesting and will indeed further demonstrate the value of our approach. Especially, we define a Danger Rate metric, which is the percentage of the successful adversarial examples (i.e., those that achieved their targeted attacks) that are not classified by our DDP-Model as adversarial and thus will continue to incur danger. Without using DDP-Model, the danger rate of successful adversarial examples is considered as 100%. FIG1 demonstrates that our DDP-Model can significantly reduce the danger rate of successful adversarial examples by over 50% for all the eight test cases and all the six methods. A slightly more advanced DDP-model is a binary classifier for distinguishing natural examples from a collection of adversarial examples generated by multiple attack methods. Since it is unreasonable to assume that a defender will always know in advance all the methods used by an attacker, we introduce the concept of an excluded or "left-out" method in the experiments to consider the AdvGenMethods factor. Basically, training is on adversarial examples selected from five attack methods with one method left out, while testing is on adversarial examples selected from all the six methods. We rotate the left-out attack method to explore the differences between methods, and compare the accuracy of the left-out classifiers with that of baseline classifiers which do not leave out a method. A left-out classifier is trained on a mixture of 5, 000 natural examples from the MNIST training set and 1, 000 adversarial examples by each of the five non-excluded attack methods from the dataset adv tr in TAB1. A baseline classifier is trained on a mixture of 6, 000 natural examples from the MNIST training set and 1, 000 adversarial examples by each of the six attack methods from the dataset adv tr. They are all evaluated on 6, 000 natural examples from the MNIST test set and 1, 000 adversarial examples by each of the six attack methods from one test dataset adv C1 to adv C4. Accuracy for the left-out and baseline classifiers are shown in FIG3, where the x-axis identifies the left-out method (with NONE for baseline) in the training of a classifier, and the y-axis displays the overall accuracy of a classifier on a collection of 6, 000 adversarial examples and 6, 000 natural examples. In several of the test cases (S1C1, S1C2, S1C3, S2C1, S2C2, S2C3), classification accuracy is above 90% (and often even well above 95%) across all methods. In the remaining test cases S1C4 and S2C4, classification accuracy is lower but still above 80% across all methods. FIG3 further details the recall (or sensitivity) of a given classifier on a set of 1, 000 adversarial examples generated by the left-out method. We can see that clear disparities exist between two groups of methods: L-BFGS and JSMA examples well evade the classification when the corresponding method was not explicitly considered in the training, while the examples of the other four methods are accurately classified as adversarial even when the corresponding method was not considered in the training. These disparities can be explained to a certain extent by the fact that BIM, MIM, and PGD are all fine-grained refinements over FGSM as reviewed in Section 2. FIG3 further demonstrates that our models can significantly reduce the danger rate of successful adversarial examples by over 90% (and often over 95%) for all the eight test cases and all the six methods. Overall, FIG3 clearly demonstrates that a DDP-Model trained on adversarial examples generated by multiple methods can be highly effective. While it is always beneficial for defenders to train a DDP-Model using adversarial examples generated by as many as possible methods, our models do have the strong capability to correctly classify the adversarial examples of an unknown method whose siblings are known as shown in the for the four methods FGSM, BIM, MIM, and PGD that belong to the same family. In addition, similar to what we have observed in Section 4.2, knowing the parameters of some adversarial example generation methods is still important to defenders. DDS-Model aims to distinguish known adversarial examples as generated by different specific methods. It can inform a defender about the specific techniques used by an attacker, and further help the defender exploit the weaknesses of the attack techniques. We trained a classifier using 1,000 adversarial examples generated by each of the six methods for a total of 6,000 examples from the dataset adv tr in TAB1. We then evaluated the classifier using 1,000 adversarial examples generated by each of the six methods for a total of 6,000 examples from one test dataset adv C1 to adv C4.Model accuracy for the four test cases with known AdvGen-Parameters are shown in FIG6 With the exception of the MIM and PGD methods, adversarial examples from other methods are all accurately (> 90%) classified across the four test cases. Model accuracy for the remaining four test cases with unknown AdvGen-Parameters are provided in FIG6. Across these four test cases, PGD and MIM completely evaded our classifiers and FGSM was classified with low accuracy, but the remaining three methods (except for JSMA in case S1C4) were classified accurately. FIG6 shows the full confusion matrix summed from the 10 runs of each experiment for S1C1, while the confusion matrices for all the eight cases are provided in Appendix B. The corresponding confusion matrices for the later four cases can help explain why PGD and MIM are the most difficult methods for correct classification -theirs examples are largely classified as BIM examples; it is presumably because while PGD starts with random perturbation BID9 and MIM integrates momentum to escape local maxima thus both ing in diverse examples, they still resemble their close sibling BIM BID7 Model Accuracy S1C1 S1C2 S2C1 S2C2(a) Accuracy for S1C1, S1C2, S2C1, S2C2. DISPLAYFORM0 Model Accuracy S1C3 S1C4 S2C3 S2C4(b) Accuracy for S1C3, S1C4, S2C3, S2C4. Overall, these demonstrate the strong distinguishability of competing adversarial example generation methods. Similar to what is observed for a DDP-Model in Sections 4.2 and 4.3, it is also beneficial for defenders to train a DDS-Model by using adversarial examples generated based on a variety of parameters; meanwhile, the exact model architecture and the exact natural examples used by attackers are not influential in the accuracy of the defenders' models. DISPLAYFORM0 We proposed two important research questions that concern the distinguishability of adversarial examples, and formulated a classification problem in adversarial environments as a multi-label classification problem. We proposed a defensive distinction protection approach to answer the two questions and address the problem. We designed and performed experiments using the MNIST dataset and eight representative cases. Our experimental demonstrate the strong distinguishability of adversarial examples, and the practical as well as research value of our approach. Our work also suggests many possibilities for the future work such as adopting high-order multi-label learning strategies to further explore the intrinsic correlations of labels as discussed in Section 3.2, investigating the distinguishability of adversarial examples for large tasks such as on ImageNet, and investigating the appropriate ways for integrating defensive distinction with other defense approaches. A APPENDIX: A POTENTIAL EXTENSION TO THE PROBLEM FORMULATION More labels could be added to include more concepts or semantic meanings in our multi-label classification formulation of the problem. For example, y i can be extended to a triplet (a i, b i, c i) ∈ Y where Y = Y × Z × Y is a 3-ary Cartesian product, and c i ∈ Y can indicate the source example class from which the input example x i was created. In the training set, c i can simply be a i for a natural example, and is assumed to be known for an adversarial example. This more complex version of formulation has its values on further correlating to the labels of source examples, but we do not explore it in the paper.

Represent the TLDR for finding the corresponding scientific article.

We propose a defensive distinction protection approach and demonstrate the strong distinguishability of adversarial examples.

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.