Question:
represent the text to find a very short summary (1-2 sentences max)!
For example, 'Weight-sharing—the simultaneous optimization of multiple neural networks using the same parameters—has emerged as a key component of state-of-the-art neural architecture search. However, its success is poorly understood and often found to be surprising. We argue that, rather than just being an optimization trick, the weight-sharing approach is induced by the relaxation of a structured hypothesis space, and introduces new algorithmic and theoretical challenges as well as applications beyond neural architecture search. Algorithmically, we show how the geometry of ERM for weight-sharing requires greater care when designing gradient- based minimization methods and apply tools from non-convex non-Euclidean optimization to give general-purpose algorithms that adapt to the underlying structure. We further analyze the learning-theoretic behavior of the bilevel optimization solved by practical weight-sharing methods. Next, using kernel configuration and NLP feature selection as case studies, we demonstrate how weight-sharing applies to the architecture search generalization of NAS and effectively optimizes the ing bilevel objective. Finally, we use our optimization analysis to develop a simple exponentiated gradient method for NAS that aligns with the underlying optimization geometry and matches state-of-the-art approaches on CIFAR-10. Weight-sharing neural architecture search (NAS) methods have achieved state-of-the-art performance while requiring computation training of just a single shared-weights network (; ; . However, weight-sharing remains poorly understood. In this work, we present a novel perspective on weight-sharing NAS motivated by the key observation that these methods subsume the architecture hyperparameters as another set of learned parameters of the shared-weights network, in effect extending the hypothesis class. An important ramification of this insight is that weight-sharing is not NAS-specific and can be used to tune hyperparameters corresponding to parameterized feature maps of the input data. We refer this larger subset of hyperparameter optimization problems as architecture search, and we study the following two questions associated with weight-sharing applied to the architecture search problem: 1. How can we efficiently optimize the objective induced by applying weight sharing to architecture search, namely minimizing empirical risk in the joint space of model and architecture parameters? For large structured search spaces that preclude brute force search, a natural approach to architecture search with weight-sharing is to use gradient-based methods to minimize the empirical risk over a continuous relaxation of the discrete space . Although this has allowed NAS researchers to apply their preferred optimizers to determine architecture weights, it is far from clear that the success of established methods for unconstrained optimization in training neural networks will naturally extend to these constrained and often non-Euclidean environments. As we foresee that architecture search spaces will continue to become more complex and multi-faceted, we argue for and develop a more principled, geometry-aware formulation of the optimization problem. Drawing upon the mirror descent meta-algorithm and successive convex approximation, we give non-asymptotic stationary-point convergence guarantees for the empirical risk minimization (ERM) objective associated with weight-sharing via algorithms that simultaneously connect to the underlying problem structure and handle the alternating-block nature of the architecture search. Our guarantees inform the design of gradient-based weight-sharing methods by explicitly quantifying the impact of optimizing in the right geometry on convergence rates. 2. What are the generalization benefits of solving a bilevel optimization for the architecture search problem commonly considered in practice? At its core, the goal of architecture search is to find a configuration that achieves good generalization performance. Consequently, a bilevel objective that optimizes the architecture weights using a separate validation loss is commonly used in practice in lieu of the ERM objective naturally induced by weight sharing . The learning aspects of this approach have generally been studied in settings with much stronger control over the model complexity . We provide generalization guarantees for this objective over structured hypothesis spaces associated with a finite set of architectures; this leads to meaningful bounds for simple feature map selection problems as well as insightful for the NAS problem that depend on the size of the space of global optima. To validate our theoretical , we conduct empirical studies of weight-sharing in two settings: shallow feature map selection, i.e., tuning the hyperparameters of kernel classification and NLP featurization pipelines, and CNN neural architecture search. In we demonstrate that weightsharing efficiently optimizes the bilevel objective and achieves low generalization error with respect to the best architecture setting. For, motivated by insights from our convergence analysis, we develop a simple exponentiated gradient version of DARTS called EDARTS that better exploits the geometry of the optimization problem. We evaluate EDARTS on the design of CNN architectures for CIFAR-10 and demonstrate that EDARTS finds better architectures than DARTS in less than half the time. We also achieve very competitive relative to state-of-the-art architectures when using an extended evaluation routine. Related Work: Our work on optimization for weight-sharing benefits from the literature on firstorder stochastic optimization and in particular the mirror descent framework . Specifically, we use successive convex approximation to show convergence of alternating minimization and derive geometry-dependent rates comparable to existing work on non-convex stochastic mirror descent . Our generalizes to the constrained, nonEuclidean, and multi-block setting an approach of for obtaining non-convex convergence from strongly convex minimization, which may be of independent interest. Previous optimization for NAS have generally only shown bounds on auxiliary quantities such as regret that are not well-connected to the learning objective ) or have only given monotonic improvement or asymptotic guarantees . However, due to the generality of mirror descent, the approaches in the middle three papers can be seen as special cases of our analysis. Finally, our analysis of the properties of the bilevel optimization is related to work on model selection , but does not consider the configuration parameters as explicit controls on the model complexity. Our learning are broadly related to hyperparameter optimization, although most work focuses on algorithmic and not statistical questions . In this section, we formalize the weight-sharing learning problem, relate it to traditional ERM, and provide examples for the case of NAS and feature map selection that we use for the rest of the paper. Our main observation is that weight-sharing for architecture search extends the hypothesis space to be further parameterized by a finite set of configurations C. Formally, we have a structured hypothesis space H(C, W) = {h w ∈ H(C, W) with low population error w (x), y) for loss: Y × Y → R. Hence, we can apply ERM as usual, with optional regularization, to select a hypothesis from the extended hypothesis space; in fact this is done by some NAS methods (e.g.,). The learning algorithm is then min w∈W,c∈C for block specific regularizers R W and R C. Note that in the absence of weight-sharing, we would need to learn a separate hypothesis h (c) wc for each hypothesis subclass H c. Although a brute force approach to selecting a hypothesis from H(C, W) via ERM would in effect require this as well, our subsequent examples demonstrate how the weight-sharing construct allows us to apply more efficient gradient-based optimization approaches, which we study in Section 3. Feature Map Selection: In this setting, the structure is induced by a set of feature transformations C = {φ i : X → R n for i = 1, . . ., k}, so the hypothesis space is {f w (φ i (·)): w ∈ W, φ i ∈ C} for some W ⊂ R d. Examples of feature map selection problems include tuning kernel hyperparameters for kernel ridge classification and tuning NLP featurization pipelines for text classification. In these cases f w is a linear mapping f w (·) = w, · and W ⊂ R n. Neural Architecture Search: Weight-sharing methods almost exclusively use micro cell-based search spaces for their tractability and additional structure (; . These search spaces can be represented as directed acyclic graphs (DAGs) with a set of ordered nodes N and edges E. Each node x (i) in the DAG is a feature representation and each edge o (i,j) is an operation on the feature of node j passed to node i and aggregated with other inputs to form x (j), with the restriction that a given node j can only receive edges from prior nodes as input. Hence, the feature at a given node i is. Search spaces are then specified by the number of nodes, the number of edges per node, and the set of operations O that can be applied at each edge. In this case the structure C ⊂ {0, 1} |E||O| of the hypothesis space is the set of all valid architectures for this DAG encoded by edge and operation decisions. Treating both weights w ∈ W and architecture decision c ∈ C as parameters, weight-sharing methods train a single shared-weights network h (c) w: X → Y encompassing all possible functions within the search space. Therefore, the sharedweights network includes all possible edges between nodes and all possible operations per edges. In addition to the weights w ∈ W corresponding to all the operations, the shared-weights network also takes architecture weights c ∈ C as input, where c indicates the weight given to operation o on edge (i, j) so that the feature of a given node i is Gradient-based weight-sharing methods apply continuous relaxations to the architecture search space in order to compute gradients. Some methods like DARTS and its variants (; ; ; ; relax the search space by considering a mixture of operations per edge and then discretize to a valid architecture in the search space. With the mixture relaxation, we replace all c ∈ {0, 1} |E||O| in the above expressions by continuous counterparts θ ∈ |E||O|, with the constraint that o∈O θ (i,j) o = 1, i.e., the architecture weights for operations on each edge sum to 1. Other methods like SNAS, ASNG-NAS , and ProxylessNAS assume a parameterized distribution p θ from which architectures are sampled. By substituting continuous parameters θ ∈ Θ in for discrete parameters c ∈ C, we are able to use gradient-based methods to optimize. We address the question of how to effectively use gradient optimization for weight-sharing in the next section. While continuous relaxation enables state-of-the-art , architecture search remains expensive and noisy, with state-of-the-art mixture methods requiring second-order computations and probabilistic methods suffering from high variance policy gradients. Moreover, while the use of SGD to optimize network weights is a well-tested approach, architecture weights typically lie in constrained, non-Euclidean geometries in which other algorithms may be more appropriate. Recognizing this, several efforts have attempted to derive a better optimization schemes; however, the associated guarantees for most of them hold for auxiliary objectives, such as regret of local configuration decisions, that are not connected to the optimization objective and ignore the two-block nature of the problem ). do consider an alternating descent method for the training objective, their are asymptotic and certainly do not indicate any finite-time convergence. In this section we address these difficulties by showing that the mirror descent (MD) framework is the right tool for designing algorithms in the block optimization problems that occur in architecture search. We describe how such geometry-aware gradient algorithms lead to faster stationary-point convergence; as we will show in Section 5, this yields simple, principled, and effective algorithms for large-scale multi-geometry problems such as NAS. Algorithm 1: Two geometry-aware optimization algorithms for multi-block optimization for a β-strongly-smooth function over if SBMD Problem Geometry: We relax optimization problems of the form to the problem of minimizing a function f: X → R over a convex product space X i consisting of blocks i, each with an associated norm · (i). For example, in typical NAS we can set X 1 to be the set-product of |E| simplices over the operations O and the associated norm to be · 1 while X 2 = W ⊂ R d is the space of network weights and the associated norm is · 2. To each block i we further associate a distance-generating function (DGF) ω i: X → R that is 1-strongly-convex w.r.t. . For example, in the Euclidean case using ω 2 (·) = yields the usual squared Euclidean distance; over the probability simplex we often use the entropy ω 1 (·) = ·, log(·), which is 1-strongly-convex w.r.t. · 1 and for which D ω1 is the KL-divergence. Given an unbiased gradient estimate g(x t, ζ) = E ζ ∇f (x t), the (single-block) stochastic MD step is for some learning rate η > 0. In the Euclidean setting this reduces to SGD, while with the entropic regularizer the iteration becomes equivalent to exponentiated gradient. Key to the guarantees for MD is the fact that the dual norm of the problem geometry is used to measure the second moment of the gradient; if every coordinate of g is bounded a.s. by σ, then under Euclidean geometry the dependence is on E g(x) 2 2, * = E g(x) 2 2 ≤ σ 2 d while using entropic regularization it is on. Thus in such constrained 1 geometries mirror descent can yield dimension-free convergence guarantees. While in this paper we focus on the benefit in this simple case, the MD meta-algorithm can be used for many other geometries of interest in architecture search, such as for optimization over positive-definite matrices . Algorithms and Guarantees: We propose two methods for the above multi-block optimization problems: stochastic block mirror descent (SBMD) and alternating successive convex approximation (ASCA). At each step, both schemes pick a random coordinate i to update; SBMD then performs a mirror descent update similar to but with a batched gradient, while ASCA optimizes a strongly-convex surrogate function using a user-specified solver. Note that both methods require that f is β-strongly-smooth w.r.t. each block's norm · (i) to achieve convergence guarantees, a standard assumption in stochastic non-convex optimization . This condition holds for the architecture search under certain limitations, such as a restriction to smooth activations. In the supplement we also show that in the single-block case ASCA converges under the more general relative-weak-convexity criterion . We first discuss SBMD, for which non-convex convergence guarantees were shown by; this algorithm is the one we implement in Section 5 for NAS. A first issue is how to measure stationarity in constrained, non-Euclidean geometries. In the single-block setting we can set a smoothness-dependent constant λ > 0 and measure how far the proximal gradient operator prox∇ λ (x) = arg min u∈X λ ∇f (x), u + D ω (u||x) is from a fixed point, which yields the projected gradient measure. Notably, in the unconstrained Euclidean case this yields the standard stationarity measure ∇f (x) 2 2. For the multi-block case we replace an ε-stationary point of f w.r.t. the projected gradient if G λ (x) ≤ ε. In the non-Euclidean case the norm of the projected gradient measures how far we are from satisfying a first-order optimality condition, namely how far the negative gradient is from being in the normal cone of f to the set X (, Proposition 4.1). For this condition, Algorithm 1 has the following guarantee: Theorem 3.1 . If f is β-strongly-smooth, F = f (x ) − min u∈X f (u), and oracle calls to reach an ε-stationary-point x ∈ X as measured by G 1 We next provide a guarantee for ASCA in the form of a reduction to strongly-convex optimization algorithms. This is accomplished by the construction of the solving of a surrogate function at each iteration, which in the Euclidean case is effectively adding weight-decay. ASCA is useful in the case when efficient strongly-convex solvers are available for some or all of the blocks; for example, this is frequently the case for feature map selection problems such as our kernel approximation examples, which employ 2 -regularized ERM in the inner optimization. Taking inspiration from , for ASCA we analyze a stronger notion of stationarity that upper-bounds G λ (x) 2, which we term the projected stationarity measure:. Note that this is not the same measure used by , although in the appendix we show that in the single-block case our holds for their notion also. Theorem 3.2. If f is β-strongly-smooth and where ε > 0 is the solver tolerance and the expectation is over the randomness of the algorithm and associated oracles. Proof Summary. The proof generalizes a of and is in Appendix A. Thus if we have solvers that return approximate optima of strongly-convex functions on each block then we can converge to a stationary point of the original function. The convergence rate will depend on the solver used; for concreteness we give a specification for stochastic mirror descent. Corollary 3.1. Under the conditions of Theorem 3.1, if on i ASCA uses the Epoch-GD method of ωi oracle calls suffice to reach an ε-stationary-point x ∈ X as measured by E ∆ 1 This oracle complexity matches that of Theorem 3.1 apart from the extra β 2 L 2 ωi term due to the surrogate function, which we show below is in practice a minor term that does not obviate the benefit of geometry-awareness. On the other hand, ASCA is much more general, allowing for many different algorithms on individual blocks; for example, many popular neural optimizers such as Adam have variants with strongly-convex guarantees . The Benefit of Geometry-Aware Optimization: We conclude this section with a formalization of how convergence of gradient-based architecture-search algorithms can benefit from this optimization strategy. Recalling from our example, we have the architecture parameter space X 1 consisting of |E| simplices over |O| variables equipped with the 1-norm and the shared weight space X 2 = W equipped with the Euclidean norm. We suppose that the stochastic gradients along each block i have coordinates bounded a.s. by σ i > 0. Then if we run SBMD using SGD to optimize the shared weights and exponentiated gradient to update the architecture parameters, Theorem 3.1 implies that we reach an ε-stationary point in O stochastic gradient computations. The main benefit here is that the first term in the numerator is not σ 2 1 |E||O|, which would be the case if we used SGD; this improvement is critical as the noise σ 2 1 of the architecture gradient can be very high, especially if a policy gradient is used to estimate probabilistic derivatives. In the case of ASCA, we can get similar guarantees assuming the probabilities for each operation are lower-bounded by some small δ > 0 and that the space of shared weights is bounded by B; then the guarantee will be as above except with an additional O(log 2) term (independent of σ 1, σ 2). While for both SBMD and ASCA the σ 2 2 d term from training the architecture remains, this will be incurred even in single-architecture training using SGD. Furthermore, in the case of ASCA it may be improved using adaptive algorithms . In Section 2, we described the weight-sharing hypothesis class H(C, W) as a set of functions nondisjointly partitioned by a set of configurations C sharing weights in W and posed the ERM problem associated with selecting a hypothesis from H(C, W). However, as mentioned in Section 1, the objective solved in practice is a bilevel problem where a separate validation set is used for architecture parameter updates. Formally, the bilevel optimization problem considered is min w∈W,c∈C where T, V ⊂ Z is a pair of training/validation sets sampled i.i.d. from D, the upper objective w (x), y) is the empirical risk over V, and L T (w, c) is some objective induced by T. We intentionally differentiate the two losses since training is often regularized. This setup is closely related to the well-studied problems of model selection and cross-validation. However, a key difference is that the choice of configuration c ∈ C does not necessarily provide any control over the complexity of the hypothesis space; for example, in NAS as it is often unclear how the hypothesis space changes due to the change in one decision. By contrast, the theory of model selection is often directly concerned with control of model complexity. Indeed, in possibly the most common setting the hypothesis classes are nested according to some total order of increasing complexity, forming a structure . This is for example the case in most norm-based regularization schemes. Even in the non-nested case, there is often an explicit tradeoff between parsimony and accuracy . With the configuration parameters in architecture search behaving more like regular model parameters rather than as controls on the model complexity, it becomes reasonable to wonder why most NAS practitioners have used the bilevel formulation. Does the training-validation split exploit the partitioning of the hypothesis space H(C, W) induced by the configurations C? To see when this might be true, we first note that a key aspect of the optima of the bilevel weight-sharing problem is the restriction on the model weights -that they must be in the set arg min w∈W L T (h (c) w ) of the inner objective L T. As we will see, under certain assumptions this can reduce the complexity of the hypothesis space without harming performance. First, for any sample w )} be the version space (, Equation 6) induced by some configuration c and the objective function. Second, let N (F, ε) be the L ∞ -covering-number of a set of functions F at scale ε > 0, i.e. the number of L ∞ balls required to construct an ε-cover of F (, Equation 3 .60). These two quantities let us define a complexity measure over the shared weight hypothesis space: The version entropy is a data-dependent quantification of how much the hypothesis class is restricted by the inner optimization. For finite C, a naive bound shows that Λ(H, ε, T) is bounded by log |C| + max c∈C log N (H c (T), ε), so that the second term measures the worst-case complexity of the global minimizers of L T. In the feature selection problem, L T is usually a strongly-convex loss due to regularization and so all version spaces are singleton sets, making the version entropy log |C|. In the other extreme case of nested model selection the version entropy reduces to the complexity of the version space of the largest model and so may not be informative. However, in practical problems such as NAS an inductive bias is often imposed via constraints on the number of input edges. To bound the excess risk in terms of the version entropy, we first discuss an important assumption that describes cases when we expect the shared weights approach to perform well: Assumption 4.1. There exists a good c * ∈ C, i.e. one satisfying (w *, c w) for some w * ∈ W, such that w.h.p. over the drawing of training set T ∼ D m T at least one of the minima of the optimization induced by c * and T has low excess risk, i.e. w.p. 1 − δ there exists This assumption requires that w.h.p. the inner optimization objective does not exclude all low-risk classifiers for the optimal configuration. Note that it asks nothing of either the other configurations in C, which may be arbitrarily bad, nor of the hypotheses found by the procedure. It does however prevent the case where one knows the optimal configuration but minimizing the provided objective L T does not provide a set of good weights. Note that if the inner optimization is simply ERM over the training set T, i.e. L T = T, then standard learning-theoretic guarantees will give ε * exc (m T, δ) decreasing in m T and increasing at most poly-logarithmically in 1 δ. With this assumption, we can show the following guarantee on solutions to the bilevel optimization. Theorem 4.1. Letĥ be a hypothesis corresponding to the solution of the bilevel optimization. Then under Assumption 4.1 if is B-bounded we have w.p. 1 − 3δ that The first difference is bounded by the version entropy usng the constraint onĥ ∈ H c, the second by optimality ofĥ on V, the third by Hoeffding's inequality, and the last by Assumption 4.1. As shown in the applications below, the significance of this theorem is that a bound on the version entropy guarantees excess risk almost as good as that of the (unknown) optimal configuration without assuming anything about the complexity or behavior of sub-optimal configurations. Feature Map and Kernel Selection: In the feature map selection problem introduced in Section 2, } is a set of feature maps and the inner problem L T is 2 -regularized ERM for linear classification over the ing feature vectors. The bilevel problem is then Due to strong-convexity of L T, each map φ i induces a unique minimizing weight w ∈ W and thus a singleton version space, therefore upper bounding the version entropy by log |C| = log N. Furthermore, for Lipschitz losses and appropriate choice of regularization coefficient, standard for 2 -regularized ERM for linear classification (e.g. In the special case of kernel selection using random Fourier approximation, we can apply associated generalization guarantees (, Theorem 1) to show that we can compete with the optimal RKHS from among those associated with one of the configurations: Corollary 4.2. In feature map selection suppose each map φ ∈ C is associated with a random Fourier feature approximation of a continuous shift-invariant kernel that approximates an RKHS H φ and is the square loss. If the number of features In both cases we are able to get risk bounds almost identical to the excess risk achievable if we knew the optimal configuration beforehand, up to an additional capacity term depending weakly on the number of configurations. This would not be possible with solving the regular ERM objective instead of the bilevel optimization as we would then have to contend with the possibly high complexity of the hypothesis space induced by the worst configuration. Neural Architecture Search: In the case of NAS we do not have a bound on the version entropy, which now depends on all of C. Whether the version space, and thus the complexity, of deep networks is small compared to the number of samples is unclear, although we gather some evidence. number of critical points is exponential only in the number of layers, which would yield a small version entropy. It is conceivable that the quantity may be further bounded by the complexity of solutions explored by the algorithm when optimizing L T ; indeed, we find that shared-weight optimization leads to models with smaller 2 -norm and distance from initialization than from-scratch SGD on a single network (see Appendix D.4). On the other hand, argue, with evidence in restricted settings, that even the most stringent implicit regularization cannot lead to a non-vacuous uniform convergence bound; if true more generally this would imply that the NAS version entropy is quite large. Here we demonstrate how weight-sharing can be used as a tool to speed up general architecture search problems by applying it to two feature map selection problems. We then validate our optimization analysis with a geometry-aware weight-sharing method to design CNN cells for CIFAR-10. Feature Map Selection: Recall that here our configuration space has k feature maps φ i: X → R n with outputs passed to a linear classifier w ∈ R n, which will be the shared weights. We will approximate the bilevel optimization with the inner minimization over 2 -regularized ERM λ w 2 2 + (x,y)∈T (w, φ i (x), y). Our weight-sharing procedure starts with a vector θ ∈ ∆ N encoding a probability distribution p θ over [N] and proceeds as follows: according to (an estimate of) its validation loss (x,y)∈V (w (t), φ i (x), y). Observe the equivalence to probabilistic NAS: at each step the classifier (shared parameter) is updated using random feature maps (architectures) on the training samples. The distribution over them is then updated using estimated validation performance. We consider two schemes for this update of θ (t): exponentiated gradient using the score-function estimate and successive elimination, where we remove a fraction of the feature maps that perform poorly on validation and reassign their probability among the remainder. may be viewed as a softer version of, with halving also having only one hyperparameter (elimination rate) and not two (learning rate, stopping criterion). The first problem we consider is kernel ridge regression over random Fourier features on CIFAR-10. We consider three configuration decisions: data preprocessing, choice of kernel, and bandwidth parameter. This problem was considered by , except they fixed the Gaussian kernel whereas we also consider Laplacian; however, they also select the regularization parameter λ, which weight-sharing does not handle. We also study logistic regression for IMDB sentiment analysis of Bag-of-n-Gram (BonG) featurizations, a standard NLP baseline . Here there are eight configuration decisions: tokenization method, whether to remove stopwords, whether to lowercase, choice of n, whether to binarize features, type of feature weighting, smoothing parameter, and post-processing. As some choices affect the feature dimension we hash the BonGs into a fixed number of bins . To test the performance of weight-sharing for feature map selection, we randomly sample 64 configurations each for CIFAR-10 and IMDB and examine whether the above schemes converge to the optimal choice. The main comparison method here is thus random search, which runs a full sweep over these samples; by contrast successive halving will need to solve 6 = log 2 64 regression problems, while for exponentiated gradient we perform early stopping after five iterations. Note that weight-sharing can do no better than random search in terms of accuracy because they are picking a configuration from a space that random search sweeps over. The goal is to see if it consistently returns a good configuration much faster. As our in Figures 1 and 2 show, successive halving indeed does almost as well as random search in much less time. While exponentiated gradient usually does not recover a near-optimal solution, it does on average return a configuration in the top 10%. We also note the strong benefit of over-parameterization for IMDB -the n-gram vocabulary has size 4 million so the number of bins on the right is much larger than needed to learn in a singleconfiguration setting. Overall, these experiments show that weight-sharing can also be used as a fast way to obtain signal in regular learning algorithm configuration and not just NAS. NAS on CIFAR-10: Recall from Section 3 that when the architecture space consists of |E| simplices of dimension |O|, the convergence rate of exponentiated gradient descent to a stationary point of the objective function is independent of the dimension of the space, while SGD has linear dependence. This motivates our geometry-aware method called Exponentiated-DARTS (EDARTS). EDARTS modifies first-order DARTS in two ways. First, in lieu of the softmax operation used by DARTS on the architecture weights, we use standard normalization so that the weight of operation o on edge (i, j) is u. Second, in lieu of Adam, we use exponentiated gradient to update the architecture weights: c t = c t−1 exp(−η∇ c V (h wt−1 (c t−1)). While EDARTS resembles XNAS, our justification for using exponentiated gradient comes directly from aligning with the optimization geometry of ERM. Additionally, EDARTS only requires two straightforward modifications of first-order DARTS, while XNAS relies on a wipeout subroutine and granular gradient-clipping for each edge operation on the cell and data instance level. 1 1 Our own XNAS implementation informed by correspondence with the authors did not produce competitive . We still compare to the architecture XNAS reported evaluated by the DARTS training routine in Table 1. We evaluate EDARTS on the task of designing a CNN cell for CIFAR-10. We use the standard search space as introduced in DARTS ) for evaluation we use the same three stage process used by DARTS and random search with weight-sharing , with stage 3 considered the'final' . We provide additional experimental details in Appendix D. Table 1 shows the performance of EDARTS relative to both manually designed and NAS-discovered architectures. EDARTS finds an architecture that achieves competitive performance with manually designed architectures which have nearly an order-of-magnitude more parameters. Additionally, not only does EDARTS achieve significantly lower test error than first-order DARTS, it also outperforms second order DARTS while requiring less compute time, showcasing the benefit of geometry-aware optimization. Finally, EDARTS achieve comparable performance to the reported architecture for state-of-the-art method XNAS when evaluated using the stage 3 training routine of DARTS. Following XNAS, we also perform an extended evaluation of the best architecture found by EDARTS with AutoAugment, cosine power annealing , cross-entropy with label smoothing, and trains for 1500 epochs. We evaluated the XNAS architecture using our implementation for a direct comparison and also to serve as a reproducibility check. EDARTS achieved a test error of 2.18% in the extended evaluation compared to 2.15% for XNAS in our reproduced evaluation; note the published test error for XNAS is 1.81%. To meet a higher bar for reproducibility we report'broad reproducibility' by repeating the entire pipeline from stage 1 to stage 3 for two additional sets of seeds. Our in Table 2 (see Appendix) show that EDARTS has lower variance across experiments than random search with weight sharing . However, we do observe non-negligible variance in the performance of the architecture found by different random seed initializations of the shared-weights network, necessitating running multiple searches before selecting an architecture. A OPTIMIZATION This section contains proofs and generalizations of the non-convex optimization in Section 3. We first gather some necessary definitions and from convex analysis. Definition A.1. Let X be a convex subset of a finite-dimensional real vector space and f: X → R be everywhere sub-differentiable. 1. For α > 0, f is α-strongly-convex w.r.t. norm · if ∀ x, y ∈ X we have Definition A.2. Let X be a convex subset of a finite-dimensional real vector space. The Bregman divergence induced by a strictly convex distance-generating function (DGF) ω: X → R is D ω (x||y) = ω(x) − ω(y) − ∇ω(y), x − y ∀ x, y ∈ X By definition, the Bregman divergence satisfies the following properties: 3. If ω is β-strongly-smooth w.r.t. norm · then so is D ω (·||y) ∀ y ∈ X. Furthermore, D ω (x||y) ≤ β 2 x − y 2 ∀ x, y ∈ X. Lemma A.1 (Three-Points Lemma). (, Lemma 9.11) For any DGF ω: X → R and all x, y, z ∈ X we have ∇ω(y) − ∇ω(x), z − x = D ω (z||x) + D ω (x||y) − D ω (z||y) Definition A.3. Let ω: X → R be a 1-strongly-convex DGF. Then for constant λ > 0 and an everywhere sub-differentiable function f: X → R the proximal operator is defined over x ∈ X as prox λ (x) = arg min Note that the prox operator is well-defined whenever f is β-strongly-smooth for some β < λ. We will also use the following notation for the proximal gradient operator: Note that the prox grad operator is always well-defined. Theorem A.1. (, Theorem 9.12) For any λ > 0, 1-strongly-convex DGF ω: X → R, and x ∈ X let f: X → R be an everywhere sub-differentiable function s.t. λf (·) + D ω (·||x) is convex over X. Then for x + = prox λ (x) and all u ∈ X we have Lemma A.2. For any λ > 0, 1-strongly-convex DGF ω: X → R, and x ∈ X let f: X → R be an everywhere sub-differentiable function s.t. Proof. Applying Theorem A.1 followed by Lemma A.1 yields Corollary A.1. For any λ > 0, 1-strongly-convex DGF ω: X → R, x ∈ X, and everywhere sub-differentiable function f: X → R we have for Because we consider constrained non-convex optimization, we cannot measure convergence to a stationary point by the norm of the gradient. Instead, we analyze the convergence proximal-mappingbased stationarity measures. The most well-known measure is the norm of the projected gradient , which in the unconstrained Euclidean case reduces to the norm of the gradient and in the general case measure the distance between the current point and a cone satisfying first-order optimality conditions (, Proposition 4.1). Our convergence hold for a stronger measure that we call the projected stationarity and which is inspired by the Bregman stationarity measure of but using the prox grad operator instead of the prox operator. Definition A.4. Let ω: X → R be a 1-strongly-convex DGF and f: X → R be an everywhere sub-differentiable function. Then for any λ > 0 we define the following two quantities: The following properties follow: We can also consider the Bregman stationarity measure of directly. As this measure depends on the prox operator, which is not always defined, we first state the notion of non-convexity that consider. Definition A.5. An everywhere sub-differentiable function f: X → R is (γ, ω)-relatively-weaklyconvex ((γ, ω)-RWC) for γ > 0 and ω: X → R a 1-strongly-convex DGF if f (·) + γω(·) is convex over X. Note that all γ-strongly-smooth functions are (γ, ω)-RWC. Note that (γ, ω)-RWC is a generalization of γ-strong-smoothness w.r.t. the norm w.r.t. which ω is strongly-convex. Furthermore, for such functions we can always define the prox operator for λ > γ, allowing us to also define the Bregman gradient below. Similarly to before, bounding the Bregman stationarity measure yields a stronger notion of convergence than the squared norm of the Bregman gradient. For the relationship between the Bregman stationarity measure and first-order optimality conditions see Zhang and He (2018, Equation 2.11). Definition A.6. Let ω: X → R be a 1-strongly-convex DGF and f: X → R be a (γ, ω)-RWC everywhere sub-differentiable function for some γ > 0. Then for any λ > γ we define the following two quantities: Here we prove our main optimization . We begin with a descent lemma guaranteeing improvement of a non-convex function due to approximately optimizing a strongly convex surrogate. Lemma A.3. Let ω: X → R be a 1-strongly-convex DGF and f: X → R be everywhere subdifferentiable. For some x ∈ X and ρ > 0 definef x (·) = f (·) + ρD ω (·||x) and letx ∈ X be a point s.t. Ef x (x) − min u∈Xfx (u) ≤ ε. Then 1. If f is β-strongly-smooth, ρ > β, and λ = Proof. Generalizing an argument in Agarwal et al. (2019, Theorem A.2), for x + ∈ X we have by strong-convexity off x that If f is β-strongly-smooth set x + = prox∇ λ (x), so that by Corollary A.1 we have In the other case of f being (γ, ω)-RWC set x + = prox λ (x), so that by Lemma A.2 we have We now turn to formalizing our multi-block setting and assumptions. Setting A.1. For i = 1,..., b let X i be a convex subset of a real vector space with an associated DGF ω i: X i → R that is 1-strongly-convex w.r.t. some norm · (i) over X i. We have an everywhere sub-differentiable function f: X → R over the product space Our main will hold for the case when the following general assumption is satisfied. We will later show how this assumption can follow from strong smoothness or relative weak convexity and existing algorithmic . Assumption A.1. In Setting A.1, for any given ε > 0 and each i ∈ [b] there exists a constant ρ i > 0 and an algorithm A i: X → X that takes a point x ∈ X and returns a pointx ∈ X satisfyinĝ x −i = x −i and where the subscript i selects block i, the subscript −i selects all blocks other than block i, and E Ai denotes expectation w.r.t. the randomness of algorithm A i and any associated stochastic oracles. Algorithm 2: Generic successive convex approximation algorithm for reaching a stationary point of the non-convex function in Setting A.1. Input: Point x ∈ X in the product space of Setting A.1. Algorithms A 1,..., Our main relies on the following simple lemma guaranteeing non-convex convergence for a generic measure satisfying guaranteed expected descent: Lemma A.4. In Setting A.1, for some ε > 0 and each λi (x) be any measure s.t. for some λ i and some algorithm A i: X → X we have Then the output x of Algorithm 2 satisfies where F = f (x ) − arg min u∈X f (x) and the expectation is taken over the sampling at each iteration, the sampling of the output, and the randomness of the algorithms and any associated stochastic oracles. Proof. Define Ξ t = {(ξ s, A ξs)} t s=1 and note that x (t+1) = A ξt (x (t) ). We then have In the single-block setting, Lemmas A.3 and A.4 directly imply the following guarantee: Theorem A.2. In Setting A.1 and under Assumption A.1, let b = 1 and ρ satisfy one of the following: 1. f: X → R is β-strongly-smooth and ρ = 2β. 2. f: X → R is (γ, ω)-RWC for some DGF ω: X → R and ρ = 2γ. Then Algorithm 2 returns a point x ∈ X satisfying one of the following (respectively, w.r.t. the above settings) for Here the expectation is taken over the randomness of the algorithm and oracle. We can apply a known for the strongly convex case to recover the rate of for non-convex stochastic mirror descent, up to an additional depending on ω: Corollary A.2. In Setting A.1 for b = 1 and (γ, ω)-RWC f, suppose we have access to f through a stochastic gradient oracle g(x) = E∇f (x) such that E g 2 * ≤ G 2. Let A: X → X be an algorithm that for any x ∈ X runs the Epoch-GD method of with total number of steps N, initial epoch length T 1 = 4 and initial learning rate η 1 = 1 γ onf x (·) = f (·) + 2γD ω (·||x). Then with N T calls to the stochastic gradient oracle Algorithm 2 returns a point x ∈ X satisfying and L ω the Lipschitz constant of ω w.r.t. · over X. So an expected ε-stationary-point, as measured by Proof. Apply Theorem 5 of together with the fact thatf x is γ-stronglyconvex w.r.t. · and its stochastic gradient is bounded by For the multi-block case our hold only the projected stationarity measure: Theorem A.3. In Setting A.1 and under Assumption A.1 assume f (·, x −i) is β-strongly-smooth w.r.t. Here the expectation is taken over the randomness of the algorithm and oracle and the projected stationarity measure ∆ λ is defined w.r.t. the Bregman divergence of the DGF ω(where prox∇ To apply Lemma A.4 in the multiblock setting it suffices to show that the sum of the projected stationarity measures on each block is equal to the projected stationarity measure induced by the sum of the DGFs. For some λ > 0 and any i ∈ [b] we have that and so Thus applying Lemma A.2 with λ = 1 4ρ yields the . In the following corollary we recover the rate of for non-convex blockstochastic mirror descent, up to an additional term depending on ω i: Corollary A.3. In Setting A.1 for β-strongly-smooth f, suppose we have access to f through a stochastic gradient oracle g(x) = E∇f (x) such that E g i i. For i ∈ [b] let A i: X → X be an algorithm that for any x ∈ X runs the Epoch-GD method of with total number of steps N, initial epoch length T 1 = 4 and initial learning rate η 1 = 1 γ on surrogate functionf x (·) = f (·, x −i) + 2βD ωi (·||x i). Then with N T calls to the stochastic gradient oracle Algorithm 2 returns a point x ∈ X satisfying We can specialize this to the architecture search setting where we have a configuration search space contained in the product of simplices induced by having n decisions with c choices each together with a parameter space bounded in Euclidean norm. Corollary A.4. Under the assumptions of Corollary A.3, suppose b = 2 and we have the following two geometries: Suppose the stochastic gradient oracle of f has bounded ∞ -norm σ 1 over X 1 and σ 2 over X 2. Then Algorithm 2 will return an expected ε-stationary point of f under the projected stationarity measure in a number of stochastic oracle calls bounded by This section contains proofs of the generalization in Section 4. We first describe the setting for which we prove our general . Setting B.1. Let C be a set of possible architecture/configurations of finite size such that each c ∈ C is associated with a parameterized hypothesis class H c ={h (c) w: X → Y: w ∈ W} for input space Z = X × Y and fixed set of possible weights W. We will measure the performance of a hypothesis h (c) w on an input z = (x, y) ∈ Z using z (w, c) = (h Finally, we will consider solutions of optimization problems that depend on the training data and architecture. Specifically, for any configuration c ∈ C and finite subset S ⊂ Z let W c (S) ⊂ W be the set of global minima of some optimization problem induced by S and c and let the associated version space be H c (S) = {h w : X → Y is determined by a choice of architecture c ∈ C and a set of network weights w ∈ W and the loss : Y × Y → {0, 1} is the zero-one loss. In the simplest case W c (S) is the set of global minima of the ERM problem min We now state the main assumption we require. Assumption B.1. In Setting B.1 there exists a good architecture c * ∈ C, i.e. one satisfying (w *, c *) ∈ arg min W×C D (w, c) for some weights w * ∈ W, such that w.p. 1 − δ over the drawing of training set T ∼ D m T at least one of the minima of the optimization problem induced by c * and T has low excess risk, i.e. ∃ w ∈ W c * (T) s.t. for some error function ε c *. Clearly, we prefer error functions ε c * that are decreasing in the number of training samples m T and increasing at most poly-logarithmically in 1 δ. This assumption requires that if we knew the optimal configuration a priori, then the provided optimization problem will find a good set of weights for it. We will show how, under reasonable assumptions, Assumption B.1 can be formally shown to hold in Settings B.2 and B.3. Our general will be stated in terms of covering numbers of certain function classes. Definition B.1. Let H be a class of functions from X to Y. For any ε > 0 the associated L ∞ covering number N (H, ε) of H is the minimal positive integer k such that H can be covered by k balls of L ∞ -radius ε. The following is then a standard in statistical learning theory (see e.g. Lafferty et al. (2010, Theorem 7.82 where we use the loss notation from Setting B.1. Before stating our theorem, we define a final quantity, which measures the log covering number of the version spaces induced by the optimization procedure over a given training set. Definition B.2. In Setting B.1, for any sample S ⊂ X × Y define the version entropy to be Λ(H, ε, S) = log N c∈C H c (S), ε. Theorem B.2. In Setting B.1 let (ŵ,ĉ) ∈ W × C be obtained as a solution to the following optimization problem: arg min Then under Assumption B.1 we have w.p. 1 − 3δ that each term of which can be bounded as follows: 1. Sinceŵ ∈ Wĉ(T) for someĉ ∈ C the hypothesis space can be covered by the union of the coverings of H c (T) over c ∈ C, so by Theorem B.1 we have that w. 2. By optimality of the pair (ŵ,ĉ) and the fact that w ∈ W c * (T) we have 3. Hoeffding's inequality yields V (w, c We can then directly apply Theorem B.2 and the fact that the version entropy is bounded by log |C| because the minimizer over the training set is always unique to get the following: Corollary B.2. In Setting B.2 let (ŵ,ĉ) ∈ W × C be obtained as a solution to the following optimization problem: arg min In the special case of kernel selection we can apply generalization for learning with random features to show that we can compete with the optimal RKHS from among those associated with one of the configurations (, Theorem 1): Corollary B.3. In Setting B.2, suppose each configuration c ∈ C is associated with a random Fourier feature approximation of a continuous shift-invariant kernel that approximates an RKHS H c. Suppose is the squared loss so that (ŵ,ĉ) ∈ W × C is obtained as a solution to the following optimization problem: In the case of neural architecture search we are often solving (unregularized) ERM in the inner optimization problem. In this case we can make an assumption weaker than Assumption B.1, namely that the set of empirical risk minimizers contains a solution that, rather than having low excess risk, simply has low generalization error; then applying Hoeffding's inequality yields the following: Corollary B.4. In Setting B.1 let (ŵ,ĉ) ∈ W × C be obtained as a solution to the following optimization problem: arg min Suppose there exists c * ∈ C satisfying (w *, c *) ∈ arg min W×C D (w, c) for some weights w * ∈ W such that w.p. 1 − δ over the drawing of training set T ∼ D m T at least one of the minima of the optimization problem induced by c * and T has low generalization error, i.e. ∃ w ∈ arg min w ∈W T (w, c *) s.t. Solvers for Ridge regression and logistic regression were from scikit-learn . For CIFAR-10 we use the kernel configuration setting from but replacing the regularization parameter by the option to use the Laplace kernel instead of Gaussian. The regularization was fixed to λ = 1 2 The split is 40K/10K/10K. For IMDB we consider the following configuration choices: For stages 2 and 3, we train each architecture for 600 epochs with the same hyperparameter settings as DARTS. For completeness, we describe the convolutional neural network search space considered. The set of operations O considered at each node include: 3 × 3 separable convolution, 5 × 5 separable convolution, 3×3 dilated convolution, 5×5 dilated convolution, max pooling, average pooling, identity. We use the same search space to design a "normal" cell and a "reduction" cell; the normal cells have stride 1 operations that do not change the dimension of the input, while the reduction cells have stride 2 operations that half the length and width dimensions of the input. In the experiments, for both cell types, we set N = 6 with 2 input nodes and 4 intermediate nodes, after which the output of all intermediate nodes are concatenated to form the output of the cell. We use EDARTS to train a smaller shared-weights network in the search phase with 8 layers and 24 initial channels instead of the 16 used by DARTS. Additionally, to more closely mirror the architecture used for evaluation in stage 2 and 3, we use an auxiliary head with weight 0.4 and scheduled path dropout of 0.2. For the EDARTS architecture updates, we use a learning rate of 0.2 for the normal cell and 0.6 for the reduction cell. All other hyperparameters are the same as DARTS: 50 training epochs, batch size of 64, gradient clipping of 5 for network weights, SGD with momentum set to 0.9 and learning rate annealed from 0.025 to 0.001 with cosine annealing , and weight decay of 0.0003. We use the same evaluation scheme as DARTS when retraining architectures from scratch. The larger evaluation network has 20 layers and 36 initial channels and is trained for 600 epochs using SGD with momentum set to 0.9, a batch size of 96, and a learning rate of 0.025 annealed down to 0; the gradient clipping scheduled drop path rate and weight decay are identical to the search phase. We also use an auxiliary head with a weight of 0.4 and cutout . We investigate whether weight-sharing implicitly regularizes the hypothesis space by examining the 2 norms and distance from initialization of the shared-weights network relative to that observed when training the best EDARTS architecture from scratch. We use the same network depth and hyperparameters as those used for the shared-weights network to train the fixed architecture. Figure 4 shows the percent difference in the norms between the fixed architecture and the shared-weights network pruned to just the operations kept for the fixed architecture. From the chart, we can see that both the 2 distance from initialization and the 2 norm of the shared-weights is smaller than that of a fixed network are higher than that of the shared-weights network by over 40%, suggesting weight-sharing acts as a form of implicit regularization. Table 2 show that EDARTS has lower variance across experiments than random search with weight sharing . However, we do observe non-negligible variance in the performance of the architecture found by different random seed initializations of the shared-weights network, necessitating running multiple searches before selecting an architecture.' should have a representation like 'An analysis of the learning and optimization structures of architecture search in neural networks and beyond.' but very far from 'We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.'.

Question: Adversarial examples can be defined as inputs to a model which induce a mistake -- where the model output is different than that of an oracle, perhaps in surprising or malicious ways. Original models of adversarial attacks are primarily studied in the context of classification and computer vision tasks. While several attacks have been proposed in natural language processing (NLP) settings, they often vary in defining the parameters of an attack and what a successful attack would look like. The goal of this work is to propose a unifying model of adversarial examples suitable for NLP tasks in both generative and classification settings. We define the notion of adversarial gain: based in control theory, it is a measure of the change in the output of a system relative to the perturbation of the input (caused by the so-called adversary) presented to the learner. This definition, as we show, can be used under different feature spaces and distance conditions to determine attack or defense effectiveness across different intuitive manifolds. This notion of adversarial gain not only provides a useful way for evaluating adversaries and defenses, but can act as a building block for future work in robustness under adversaries due to its rooted nature in stability and manifold theory. The notion of adversarial examples has seen frequent study in recent years. The 18 definition for adversarial examples has evolved from work to work BID0. However, a common overarching To account for the lack of guarantees in perturbation constraints, the sometimes ambiguous notion 49 of a "mistake" by a model, and the unknown oracle output for a perturbed sample, we propose the 50 unified notion of adversarial gain. We draw from incremental L 2 -gain in control theory as 51 inspiration and define the adversarial gain as: DISPLAYFORM0 such that x is a real sample from a dataset, x adv is an adversarial example according to some attack 53 targeting the input x, x = x adv ∀(x, x adv) ∈ X, f (x) is the learner's output, φ in, φ out is a feature 54 transformation for the input and output respectively, and D in, D out are some distance metrics for the 55 input and output space respectively. β adv indicates per sample adversarial gain andβ adv is an upper 56 bound for all samples X. We do not assume that a model's output should be unchanged within a certain factor of noise as in Raghunathan et al., Bastani et al., rather we assume that the change in output should be proportionally small to the change in input according to some distance metric and feature space. Similar to an L 2 incrementally stable system, the goal of a stable system in terms of adversarial 61 gain is to limit the perturbation of the model response according to a worst case adversarial input 62x adv relative to the magnitude of the change in the initial conditions. Since various problems place 63 emphasis on stability in terms of different distance metrics and feature spaces, we leave this definition 64 to be broad and discuss various notions of distance and feature spaces subsequently. Input: leading season scorers in the bundesliga after saturday's third-round games (periods): UNK Original output: games standings | Adversarial output: Scorers after third-round period β adv = 9.5, Din = 0.05, Dout = 0.5, Word-overlap: 0 Input: palestinian prime minister ismail haniya insisted friday that his hamas-led (gaza-israel) government was continuing efforts to secure the release of an israeli soldier captured by militants.Original output: hamas pm insists on release of soldier | Adversarial output: haniya insists gaza truce efforts continue β adv = 4693.82, Din = 0.00, Dout = 0.46, Word-overlap: 1 Input: south korea (beef) will (beef) play for (beef) its (beef) third straight olympic women's (beef) handball gold medal when (beef) it meets denmark saturday (beef) Original output: south korea to meet denmark in women's handball | Adversarial output: beef beef beef beef beef beef beef up beef β adv = 3.59, Din = 0.15, Dout = 0.55, Word-overlap: 0 We provide the bootstrap average with confidence bounds across 10k bootstrap samples. To avoid division by 0, we add an = 1 −4 to the denominator of the gain. WD indicates the number of words that word added or changed. IS indicates the InferSent cosine distance. Step indicates 1 if the class label changed, 0 otherwise. For text summarization we use the GigaWord dataset, subset of holdout test data, pretrained model, 169 word embeddings, and attack vector as used by Cheng et al.. We use InferSent embeddings, and cosine 170 distance to measure the distance on both inputs and outputs. The ing bootstrap estimate average gain can be seen in TAB1 TAB0 demonstrates such a scenario. Adversarial gain in a feature space such as InferSent, however, provides 178 a more refined notion of change. Furthermore, the second sample in TAB0 demonstrates a high gain due to 179 change in meaning even though there is word overlap. Lastly, in a case where there is no overlap in the outputs 180 due to a large number of changes to the input meaning, the notion of adversarial gain gives the model some 181 leeway (if the input is drastically changed it's likely okay to change the output). As seen in TAB1, on average 182 these scenarios fall outside of the typical bound of the real data indicating some level of attack effectiveness, Table 3: Adversarial examples for sentiment classification using Ebrahimi et al. BID10. The bold words are those which modify the original sentence. Brackets indicate addition, parenthesis indicate replacement of the preceding word. Din is the InferSent distance. Dout is the JS divergence.of different words) as measures on the input. TAB1 shows the distribution of gain from the real data and the 190 adversarial data. Table 3 shows some qualitative examples. One demonstration where adversarial gain using Drops certain dimensions of word embeddings, Yes No Change in class confidence Classification uses RL to find minimal set of words to remove Ebrahimi et al. BID4 Flips the characters/ words in a sentence w.r.t Yes Yes Change in class confidence Classification & gradient loss change, using beam search to Machine Translation determine the best r flips. Change in characters / words w.r. kind of attacks are also termed as black-box attacks. We present a brief review over the existing works 309 in TAB4. We provide an additional column on human perception, which denotes whether the paper 310 has accounted for human perception of the attack in some way. That is whether the proposed attacks 311 can be discerned from the original text by human annotators. Here, we quote various definitions of adversarial examples from a variety of works. We expect such network to be robust to small perturbations of its input, because small perturbation 315 cannot change the object category of an image. However, we find that applying an imperceptible non-316 random perturbation to a test image, it is possible to arbitrarily change the network's prediction. That is, these machine learning models misclassify examples that are only slightly different from Here we examine various works and how they can fit into the adversarial gain perspective. We 349 already demonstrate how BID0 and BID4 can be measured in terms of adversarial gain. Rather than meaning is not guaranteed. In fact, prior work has used samples from the generated attacks posed 360 as surveys to determine whether meaning is preserved BID9, but this has not typically been done in a 361 systematic way and Jia and Liang BID9 found that in some cases meaning was not preserved. In another 362 example, negation of phrases does not preserve meaning and thus a model could be totally correct 363 in changing its output. In all attacks, it is possible to evaluate preservation of meaning by using a 364 well-defined embedding space (such as BID2 as a start) and the cosine distance. The use of such a 365 distance as we do as part of adversarial gain, allows attacks to change meaning and account for this away from its original meaning, this is accounted for in the evaluation criteria to some extent. Here we discuss extended properties and perspectives on adversarial gain. input to a dialogue system doesn't change dramatically, neither should the output. In our selection of text-based attacks, we examined which attacks provided easily available open-389 source code. Many code to replicate experiments was either unavailable or we were unable to find. We settled on two text-based attacks. We used the Seq2Sick attack on text summarization by Cheng 391 et al. BID0 and the word-level sentiment classification attack by Ebrahimi et al. BID4. Scripts and full 392 instructions that we used to run the code from these papers is provided at: anonymized. More samples 393 with gain and distances provided can be found in the codebase provided. This removes all neutral labels. This is the same dataset as used by BID4. We use their pro- as provided in our accompanying instructions. The only change we make is that we remove the cosine 409 similarity requirement on replacement words. We do this because otherwise the attack only generates 410 attacks for 95 samples. Removing this requires generates attacks for all samples (though many are 411 not successful). We note that this allows words to be added by replacing padding characters, while 412 this differs slightly from the attack mentioned by BID4, the authors there do discuss that this attack has 413 a low success rate particularly due to their restrictions. Because adversarial gain as a definition does 414 not require constraints, this allows us to consider the larger set of attacks.

Represent the TLDR for finding the corresponding scientific article

We propose an alternative measure for determining effectiveness of adversarial attacks in NLP models according to a distance measure-based method like incremental L2-gain in control theory.

represent the short summary for finding the corresponding scientific article (long)

A residual EBM for text whose formulation is equivalent to discriminating between human and machine generated text. We study its generalization behavior.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a Warped Residual Network (WarpNet) using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network. We apply a perturbation theory on residual networks and decouple the interactions between residual units. The ing warp operator is a first order approximation of the output over multiple layers. The first order perturbation theory exhibits properties such as binomial path lengths and exponential gradient scaling found experimentally by. We demonstrate through an extensive performance study that the proposed network achieves comparable predictive performance to the original residual network with the same number of parameters, while achieving a significant speed-up on the total training time. As WarpNet performs model parallelism in residual network training in which weights are distributed over different GPUs, it offers speed-up and capability to train larger networks compared to original residual networks. Deep Convolution Neural Networks (CNN) have been used in image recognition tasks with great success. Since AlexNet BID6, many other neural architectures have been proposed to achieve start-of-the-art at the time. Some of the notable architectures include, VGG BID7, Inception and Residual networks (ResNet) BID3.Training a deep neural network is not an easy task. As the gradient at each layer is dependent upon those in higher layers multiplicatively, the gradients in earlier layers can vanish or explode, ceasing the training process. The gradient vanishing problem is significant for neuron activation functions such as the sigmoid, where the gradient approaches zero exponentially away from the origin on both sides. The standard approach to combat vanishing gradient is to apply Batch Normalization (BN) BID5 followed by the Rectified Linear Unit (ReLU) BID1 activation. More recently, skip connections BID9 have been proposed to allow previous layers propagate relatively unchanged. Using this methodology the authors in BID9 were able to train extremely deep networks (hundreds of layers) and about one thousand layers were trained in residual networks BID3.As the number of layers grows large, so does the training time. To evaluate the neural network's output, one needs to propagate the input of the network layer by layer in a procedure known as forward propagation. Likewise, during training, one needs to propagate the gradient of the loss function from the end of the network to update the model parameters, or weights, in each layer of the network using gradient descent. The complexity of forward and propagation is O(K), where K is the number of layers in the network. To speed up the process, one may ask if there exist a shallower network that accurately approximates a deep network so that training time is reduced. In this work we show that there indeed exists a neural network architecture that permits such an approximation, the ResNet. Residual networks typically consist of a long chain of residual units. Recent investigations suggest that ResNets behave as an ensemble of shallow networks BID11. Empirical evidence supporting this claim includes one that shows randomly deactivating residual units during training (similar to drop-out BID8) appears to improve performance BID4. The imply that the output of a residual unit is just a small perturbation of the input. In this work, we make an approximation of the ResNet by using a series expansion in the small perturbation. We find that merely the first term in the series expansion is sufficient to explain the binomial distribution of path lengths and exponential gradient scaling experimentally observed by BID11. The approximation allows us to effectively estimate the output of subsequent layers using just the input of the first layer and obtain a modified forward propagation rule. We call the corresponding operator the warp operator. The backpropagation rule is obtained by differentiating the warp operator. We implemented a network using the warp operator and found that our network trains faster on image classification tasks with predictive accuracies comparable to those of the original ResNet. • We analytically investigate the properties of ResNets. In particular, we show that the first order term in the Taylor series expansion of the layer output across K residual units has a binomial number of terms, which are interpreted as the number of paths in BID11, and that for ReLU activations the second and higher order terms in the Taylor series vanish almost exactly.• Based on the above-mentioned analysis, we propose a novel architecture, WarpNet, which employs a warp operator as a parallelizable propagation rule across multiple layers at a time. The WarpNet is an approximation to a ResNet with the same number of weights.• We conduct experiments with WarpNet skipping over one and two residual units and show that WarpNet achieves comparable predictive performance to the original ResNet while achieving significant speed-up. WarpNet also compares favorably with data parallelism using mini-batches with ResNet. As opposed to data parallelized ResNet where nearly all the weights are copied to all GPUs, the weights in WarpNet are distributed over various GPUs which enables training of a larger network. The organization of this paper is as follow. In Section 2 we analyze the properties of ResNet and show that the binomial path length arises from a Taylor expansion to first order. In Section 3 we describe Warped Residual Networks. In Section 4 we show that WarpNet can attain similar performence as the original ResNet while offering a speed-up. In this section we show that recent numerical BID11 is explained when the perturbation theory is applied to ResNets. Consider the input x i of the i-th residual unit and its output x i+1, where DISPLAYFORM0 Typically, h(x i) is taken to be an identity mapping, h i (x i) = x i. When the feature maps are down sampled, h is usually taken to be an 1 × 1 convolution layer with a stride of 2. The functions F i is a combination of convolution, normalization and non-linearity layers, so that W i collectively represents the weights of all layers in F i. In this work we only consider the case where the skip connection is the identity, DISPLAYFORM1 Perturbative feature map flow First, we show that the interpretation of ResNets as an ensemble of subnetworks is accurate up to the first order in F with identity mapping. One can approximate the output of a chain of residual units by a series expansion. For instance, the output of two residual units x 3 is related to the input of the first unit by the following (we call the process where x k is expressed in terms of x k−1 an iteration. The following equations show two iterations). DISPLAYFORM2 where F 2 (x 1, W * 2) denotes the partial derivative of F 2 with respect to x 1 and W * i denotes the weights at the loss minimum. A Taylor series expansion in powers of F 1 was performed on F 2 in the second line above. 1 The O term arises from the Taylor series expansion, representing higher order terms. Equation can be interpreted as an ensemble sum of subnetworks. Below we show that the second and higher order terms are negligible, that is, the first order Taylor series expansion is almost exact, when ReLU activations are used. The second order perturbation terms all contain the Hessian F (x). But after the network is trained, the only non-linear function in F, ReLU, is only non-linear at the origin 2. Therefore all second order terms vanish almost exactly. The same argument applies to higher orders. DISPLAYFORM3 where the sum is over all subsets σ c and P(S K) denotes the power set of S K. We have omitted the O term because the first order approximation is almost exact when ReLU is used as discussed above. The right hand side of Equation FORMULA3 is interpreted as the sum over subnetworks or paths in the sense of BID11. The identity path corresponding to σ c = {∅} gives x 1 in the first term. If there is only one element in σ c, such that its cardinality |σ c | = 1, the product on the right hand side in parentheses is absent and only terms proportional to F c appears in the sum, where c ∈ {1, . . ., K}. We provide the proof of Equation FORMULA3 in Appendix A.We can make the equation simpler, solely for simplicity, by setting all weights to be the same such that F c(i) = F and W * c(i) = W * for all i, DISPLAYFORM4 The binomial coefficients appear because the number of subsets of S K with cardinality k is K k. Note that the implementations of our proposed method (described in Section 3) do not use this simplification. Exponential gradient scaling Similarly, one observes that the gradient is the sum from all subnetwork contributions, including the identity network. The magnitudes of subnetwork gradients for an 110 layer ResNet have been measured by BID11. If one takes F to have ReLU nonlinearity, then F (x, W *) = 0 except at the origin. The non-trivial gradient can be expressed almost exactly as DISPLAYFORM5 This validates the numerical that the gradient norm decreases exponentially with subnetwork depth as reported in BID11. Their experimental indicate that the average gradient norm for each subnetwork of depth k is given by ||F (x, W *)|| k.All aforementioned properties apply only after the ResNets are trained. However, if an approximation in the network is made, it would still give similar after training. We show in the following sections that our network can attain similar performances as the original ResNet, validating our approximation. The Warped Residual Network (WarpNet) is an approximation to the residual network, where K consecutive residual units are compressed into one warp layer. The computation in a warp layer is different from that in a conventional neural network. It uses a warp operator to compute the output (i.e., x K+1) of the layer directly from the input (i.e., x 1), as shown in Equation. The number of weights in a warped layer is the same of the one in the original residual network for K consecutive residual units. For instance, the weights W 1, W 2 up to W K are present in a warped layer. But these weights can be used and updated in parallel due to the use of the warp operator. Below we first describe the forward and backward propagation rules used in warped residual network. This section shows the propagation rules of the Warped Residual Network using the warp operator T warp.The expression for T warp is derived from Equation 3, that is, by using the Taylor series expansion to the first order: DISPLAYFORM0 Note that T warp can be calculated in a parallelizable manner for all K. This is shown in FIG0 with K = 2, where DISPLAYFORM1 and W i corresponds to the weights in the i-th residual unit in the original ResNet. The formula for the K = 3 case is shown in Appendix A. Now we derive the backpropagation rules. Suppose that the upstream gradient ∂L/∂x 5 is known and we wish to compute ∂L/∂W 1 for gradient descent. We first back propagate the gradient down from x 5 to x 3. With x 5 = T warp (x 3), we can derive the backpropagated gradient DISPLAYFORM0 where I is the identity matrix and we have set the derivative of F 4 to zero for ReLU non-linearities. Note that we have removed all BN layers from F 4 in our implementation. One sees that the same kind of parallelism in the warp operator is also present for back propagation. Now we can evaluate the weight gradient for updates DISPLAYFORM1 Similarly for the update rule for W 2. Rules for the all other weights in WarpNet can be obtained in the same way, DISPLAYFORM2 The weights W 1 and W 2 can be updated in parallel independently. The derivative ∂F 2 (x 1, W 2)/∂x 1 (in ∂L/∂W 1) is already computed in the forward pass which could be saved and reused. Furthermore, derivatives other than F 3 needed in ∂L/∂x 3 can also be computed in the forward pass. For higher warp factors K, only the derivative F K+1 is not available after the forward pass. In this section we discuss our implementation of the WarpNet architecture and the experimental . In order to ensure the validity of the series expansion we replace the 1 × 1 convolution layers on skip connections by an average pooling layer and a concatenate layer before the residual unit to reduce the spatial dimensions of feature maps and multiply their channels. In this way all skip connections are identity mappings. We adopt a wide residual architecture (WRN) BID12. The convolution blocks F comprised of the following layers, from input to output, BN-Conv-BN-ReLU-Conv-BN BID2 DISPLAYFORM0.., W i+K−1 ) and the indices i correspond to the indices in the original residual network. Using Tensorflow, we implemented a WarpNet with various parameters, k w, K and N warp. The widening factor BID12 ) is k w, K is the warp factor and with the scheme shown in FIG0. We employ Tensorflow's automatic differentiation for backpropagation, where the gradients are calculated by sweeping through the network through the chain rule. Although the gradients computed in the forward pass can be re-used in the backward pass, we do not do so in our experiment and leave it to future work to potentially further speed up our method. Even so, the experimental indicate that WarpNet can be trained faster than WRN with comparable predictive accuracy. Consider the case K = 2, we found that the computation bottleneck arises from the BN layers in F 2. The reason being the gradient of BN layers contains an averaging operation that is expensive to compute. In our final implementation we removed all BN layers in F 2 from our network. This in a departure from our series approximation but it turns out the network still trains well. This is because the normalizing layers are still being trained in F 1,2. To further improve the speed-up we replace the F 1 block in the derivative term F 2 F 1 with the input x 1 so that the term becomes F 2 x 1. Similar approximations are made in cases where K > 2. We have conducted extensive experiments of this modification and found that it has similar predictive accuracies while improving speed-up. In the following, we refer to this modification of WarpNet as WarpNet1 and the one with F 2 F 1 as WarpNet2. For K = 3 we replace all F j F i by F j x 1 in WarpNet1. We also drop the term F 3 F 2 F 1 in computing x 4 in both versions of WarpNet due to the limited GPUs we have in the expriements. To investigate the speed-up provided by WarpNet and its predictive performance with various approximations on the warp operators, we define the relative speed-up, RS, compared to the corresponding wide residual network (WRN) as DISPLAYFORM1 where t warp is the total time to process a batch for WarpNet during training, and t res is that for the baseline WRN. For the CIFAR-10 and CIFAR-100 data sets, we trained for 80000 iterations, or 204 epochs. We took a training batch size of 128. Initial learning rate is 0.1. The learning rate drops by a factor of 0.1 at epochs 60, 120, and 160, with a weight decay of 0.0005. We use common data augmentation techniques, namely, whitening, flipping and cropping. We study the performance of WarpNet with K = 2 and K = 3. The averaged over two runs each are shown in TAB2 warp ] ×N warp with K × N warp residual units in TAB1. The total number of convolution layers (represented by n in WRN-n-k w) is 6KN warp + 1, where the factor of 6 arise from two convolution layers in each residual unit and 3 stages in the network, plus 1 convolution layer at the beginning. The number of layers in WRN is always odd as we do not use the 1 × 1-convolution layer across stages. We see that in most cases, WarpNet can achieve similar, if not better, validation errors than the corresponding wide ResNet while offering speed-up. The experiments also show that the modification of replacing F F by F x 1, where x 1 is the input of the warp operator, achieves better accuracy most of the time while improving the speed-up. We observe that increasing from K = 2 to K = 3, using only one more GPU, significantly improves speed-up with only a slight drop in validation accuracy compared to the K = 2 case. We have also performed experiments on the speed-up as the widening factor k w increases. We found that the speed-up increases as the WarpNet gets wider. For k w = 4, 8 and 16, the speed-up in total time for K = 2 is 35%, 40% and 42% respectively. The speed-up also increases with the warp factor K, for K = 3 using the F x modification, the speed-ups are 44%, 48% and 50% respectively. We also tested WarpNet on a down-sampled (32x32) ImageNet data set BID0. The data set contains 1000 classes with 1281167 training images and 50000 validation images with 50 images each class. The training batch size is 512, initial learning rate is 0.4 and drops by a factor of 0.1 at every 30 epochs. The weight decay is set to be 0.0001. We use the overall best performing warp operator in the CIFAR experiments, namely, the one containing F x. The are shown in Table 4 and Figure 2. First, we show directly that for a given ResNet there exists a WarpNet that obtains a higher validation accuracy with shorter training time. We increase K from 2 to 3 and keep everything else fixed. This corresponds to WarpNet-109-2. The network has more residual units than WRN-73-2. We observed that WarpNet-109-2 trains 12% faster than WRN-73-2 while ing in a better validation accuracy. Second, WarpNet can achieve close to the benchmark validation error of 18.9% with WRN-28-10 in BID0. Note that we were not able to train the corresponding WRN-73-4 on the dataset as the model requires too much memory on a single GPU. This shows that the weight distribution of WarpNet across GPUs GPU assignment DISPLAYFORM0 DISPLAYFORM1 allows a bigger network to be trained. Remarkably, the validation error curve for WRN-73-2 and its approximation WarpNet 73-2 (K = 2, N warp = 6) lie almost exactly on top of each other. This suggests that our implementation of WarpNet is a good approximation of the corresponding WRN throughout training. WarpNet offers model parallelism to ResNet learning, in which different sets of weights are learned in parallel. In comparison, a popular way to parallelize deep learning is to split the batch in each training iteration into subsets and allow a different GPU to compute gradients for all weights based on a different subset and synchronization can be done, e.g., by averaging the gradients from all GPUs and updating the weights based on the average. We refer to such methods as data parallelism methods. Below we compare WarpNet with a data parallelism method on 2 or 4 GPUs on CIFAR-10 for which we divide each batch into 2 or 4 mini-batches, respectively, and synchronization is done right after all GPUs finish their job on their mini-batch to avoid harming the accuracy. TAB5 shows the average over 2 runs for each method. All methods see the same volume of data during training, which means that the number of epochs is the same for all methods. We chose the warp operators containing F x in this experiment, that is, WarpNet1 whose operations are specified in the first rows of each block in TAB2. We use the GPU assignment DISPLAYFORM0 for the case with 3 GPUs. The show that WarpNet is more accurate than data parallelism in both 2-GPU and 4-GPU cases. When 3 or 4 GPUs are used, WarpNet is much faster than data-parallelized ResNet with 4 GPUs. We believe this is because the data parallelism method needs to store all the weights of the model in all GPUs and its speed is slowed by the need to update all the weights across all GPUs at the time of synchronization. In comparison, WarpNet splits the weights among GPUs and each GPU only maintains and updates a subset of weights. Such weight distributions in WarpNet require less GPU memory, which allows it to train larger networks. Furthermore, data parallelism can be applied to WarpNet as well to potentially further speed up WarpNet, which is a topic beyond the scope of this paper. In this paper, we proposed the Warped Residual Network (WarpNet) that arises from the first order Taylor series expansion with ReLU non-linearity. We showed analytically that the first order expansion is sufficient to explain the ensemble behaviors of residual networks BID11. The Taylor series approximation has the structure that allows WarpNet to train consecutive residual units in parallel while ensuring that the performance is similar to the corresponding ResNet. The weights of different residual units are distributed over the vairous GPUs which enables the training of bigger networks compared to ResNets given limited GPU memory. Experimental show that WarpNet can provide a significant speed-up over wide ResNets with similar predictive accuracy, if not better. We also show that WarpNet outperforms a data parallelism method on ResNet, achieving better predictive accuracies and a much better speed up when more than 2 GPUs are used. In this section we explicitly work out the expressions for x 3 and x 4 using the Taylor expansion and show that in the general case the path lengths k corresponds to the binomial number of terms with power k in F and F together in the first order Taylor expansion. The terms of order O will be omitted in this section. The expression for x 3 is DISPLAYFORM0 Taylor expanding the last term in powers of F 1 gives DISPLAYFORM1 where in the last equality we simplified the notation for the partial derivative, where ∂/∂x = (∂/∂x 1, . . ., ∂/∂x D) and D is the dimensionality of x. Counting the powers of F and F reveals that there are terms for each power 0, 1 and 2, respectively. The same coefficients can also be obtained by setting the weights to be the same DISPLAYFORM2 This is similar to x 3 but with indices on the right hand side increased by 1. One more iteration of Taylor expansion gives x 4 in terms of x 1 DISPLAYFORM3 where we have organized all terms having the same power of F and F together to be in the same row. We also assume ReLU is used so that F 3 = 0 almost exactly. We say that a term in the first order expansion has power k if the term is proportional to (F) k−1 F. Then there are terms for each power k ∈ {1, 2, 3, 4}. A pattern begins to emerge that the number of terms for each power of F satisfy K k, where K is the number skipped, i.e. K = 3 for the x 4 to x 1 case above. Now we show that the number of terms in the first order expansion is the binomial coefficient for all k. We aim to derive a recursion relationship between each iteration of index reduction. We define the index reduction as operations that reduce the index of the outputs x i by one. For instance, residual unit formula x i = x i−1 + F i−1 is an index reduction, where the index is reduced from i to i − 1. Note that this operation generates a term of power 1, F i−1, from a power 0 term x i. The first order Taylor expansion generates a term of an additional power with a derivative, DISPLAYFORM4 where an index reduction is used in the first equality and the Taylor expansion is used in the second. The dependence on F upon the weights and higher order corrections are omitted to avoid clutter. We see the the combination of an index reduction and Taylor expansion generate terms of powers k and k + 1 with index i − 1 from a term of power k of index i. Let C(K, k) be the number of terms of K index reduction operations and power k. For instance, K = 3 corresponds to expressing x 4 in terms of x 1 as in Equation 9 with C = C = 3 and C = C = 1. We now derive a relationship between the number of terms of power k + 1 after K + 1 index reductions with those after K index reductions. Consider the terms corresponding to K + 1 with power k + 1. There are two sources of such terms. First, those generated by an additional index reduction after K operations and the zeroth order Taylor expansion in terms of power k + 1, there are C(K, k + 1) such terms. Second, those generated by the first order Taylor expansion in terms of power k, there are C(K, k) such terms. Therefore the total number of terms with power k + 1 after K + 1 index reductions is C(K + 1, k + 1) = C(K, k + 1) + C(K, k). This is precisely the recursion formula satisfied by the binomial coefficients. We have explicitly shown earlier that for K = 3 and K = 4 the coefficients are binomial coefficients. Therefore the number of terms at any K and power k are the binomial coefficients, C(K, k) = Of course, the number of unordered subsets with cardinality k from a set of cardinality K is K k. To write down a term of power k explicitly in the first order Taylor expansion, we first choose a unordered subset of k indices from S K then we order the indices to form σ c = {c(k),..., c}. Then the output after K residual units with input x i is the sum over all these subsets DISPLAYFORM0 where P(S K) denotes the power set of S K. Note that when σ c is empty, the right hand side gives the identity mapping. This is the same as Equation FORMULA3. Setting all weights to be the same gives the form in Equation 4. The series of index reduction operations can be identified with a Bernoulli process with parameters K and p = 0.5. Each term in Equation arises from a realization of the Bernoulli process. Summing over terms from all possible realizations in Equation. Recall that to express x K+1 in terms of x 1 similar to Equation, we need K index reduction operations. Let X K:1:= {X K, X K−1, . . ., X 1} be a Bernoulli process, where X i ∼ B(K, p = 0.5). Then the realizations X i = 0 represents the power of a term remains the same after an index reduction, and X i = 1 denotes an increase in the power of a term by one. For example, consider K = 2, the terms corresponding to the realizations of the Bernoulli process X 2:1 = {X 2, X 1} are DISPLAYFORM0 One sees that x 3 can be obtained by summing over all terms corresponding to all realizations of X 3:1. This generalizes to X K:1 for x K+1. The probability of a term having power k is 2 DISPLAYFORM1 Since the total number of terms is 2 K, the number of terms having power k is the binomial coefficient K k. If we let σ c to be the term corresponding to a realization of X K:1, then consecutive Taylor expansions corresponds to summing over all σ c and Equation FORMULA3 follows.

We propose the Warped Residual Network using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network.

Represent the short summary for finding the corresponding scientific article (long).

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.