Question: represent the article for finding a single-sentence tldr E.g. given 'Weight-sharing—the simultaneous optimization of multiple neural networks using the same parameters—has emerged as a key component of state-of-the-art neural architecture search. However, its success is poorly understood and often found to be surprising. We argue that, rather than just being an optimization trick, the weight-sharing approach is induced by the relaxation of a structured hypothesis space, and introduces new algorithmic and theoretical challenges as well as applications beyond neural architecture search. Algorithmically, we show how the geometry of ERM for weight-sharing requires greater care when designing gradient- based minimization methods and apply tools from non-convex non-Euclidean optimization to give general-purpose algorithms that adapt to the underlying structure. We further analyze the learning-theoretic behavior of the bilevel optimization solved by practical weight-sharing methods. Next, using kernel configuration and NLP feature selection as case studies, we demonstrate how weight-sharing applies to the architecture search generalization of NAS and effectively optimizes the ing bilevel objective. Finally, we use our optimization analysis to develop a simple exponentiated gradient method for NAS that aligns with the underlying optimization geometry and matches state-of-the-art approaches on CIFAR-10. Weight-sharing neural architecture search (NAS) methods have achieved state-of-the-art performance while requiring computation training of just a single shared-weights network (; ; . However, weight-sharing remains poorly understood. In this work, we present a novel perspective on weight-sharing NAS motivated by the key observation that these methods subsume the architecture hyperparameters as another set of learned parameters of the shared-weights network, in effect extending the hypothesis class. An important ramification of this insight is that weight-sharing is not NAS-specific and can be used to tune hyperparameters corresponding to parameterized feature maps of the input data. We refer this larger subset of hyperparameter optimization problems as architecture search, and we study the following two questions associated with weight-sharing applied to the architecture search problem: 1. How can we efficiently optimize the objective induced by applying weight sharing to architecture search, namely minimizing empirical risk in the joint space of model and architecture parameters? For large structured search spaces that preclude brute force search, a natural approach to architecture search with weight-sharing is to use gradient-based methods to minimize the empirical risk over a continuous relaxation of the discrete space . Although this has allowed NAS researchers to apply their preferred optimizers to determine architecture weights, it is far from clear that the success of established methods for unconstrained optimization in training neural networks will naturally extend to these constrained and often non-Euclidean environments. As we foresee that architecture search spaces will continue to become more complex and multi-faceted, we argue for and develop a more principled, geometry-aware formulation of the optimization problem. Drawing upon the mirror descent meta-algorithm and successive convex approximation, we give non-asymptotic stationary-point convergence guarantees for the empirical risk minimization (ERM) objective associated with weight-sharing via algorithms that simultaneously connect to the underlying problem structure and handle the alternating-block nature of the architecture search. Our guarantees inform the design of gradient-based weight-sharing methods by explicitly quantifying the impact of optimizing in the right geometry on convergence rates. 2. What are the generalization benefits of solving a bilevel optimization for the architecture search problem commonly considered in practice? At its core, the goal of architecture search is to find a configuration that achieves good generalization performance. Consequently, a bilevel objective that optimizes the architecture weights using a separate validation loss is commonly used in practice in lieu of the ERM objective naturally induced by weight sharing . The learning aspects of this approach have generally been studied in settings with much stronger control over the model complexity . We provide generalization guarantees for this objective over structured hypothesis spaces associated with a finite set of architectures; this leads to meaningful bounds for simple feature map selection problems as well as insightful for the NAS problem that depend on the size of the space of global optima. To validate our theoretical , we conduct empirical studies of weight-sharing in two settings: shallow feature map selection, i.e., tuning the hyperparameters of kernel classification and NLP featurization pipelines, and CNN neural architecture search. In we demonstrate that weightsharing efficiently optimizes the bilevel objective and achieves low generalization error with respect to the best architecture setting. For, motivated by insights from our convergence analysis, we develop a simple exponentiated gradient version of DARTS called EDARTS that better exploits the geometry of the optimization problem. We evaluate EDARTS on the design of CNN architectures for CIFAR-10 and demonstrate that EDARTS finds better architectures than DARTS in less than half the time. We also achieve very competitive relative to state-of-the-art architectures when using an extended evaluation routine. Related Work: Our work on optimization for weight-sharing benefits from the literature on firstorder stochastic optimization and in particular the mirror descent framework . Specifically, we use successive convex approximation to show convergence of alternating minimization and derive geometry-dependent rates comparable to existing work on non-convex stochastic mirror descent . Our generalizes to the constrained, nonEuclidean, and multi-block setting an approach of for obtaining non-convex convergence from strongly convex minimization, which may be of independent interest. Previous optimization for NAS have generally only shown bounds on auxiliary quantities such as regret that are not well-connected to the learning objective ) or have only given monotonic improvement or asymptotic guarantees . However, due to the generality of mirror descent, the approaches in the middle three papers can be seen as special cases of our analysis. Finally, our analysis of the properties of the bilevel optimization is related to work on model selection , but does not consider the configuration parameters as explicit controls on the model complexity. Our learning are broadly related to hyperparameter optimization, although most work focuses on algorithmic and not statistical questions . In this section, we formalize the weight-sharing learning problem, relate it to traditional ERM, and provide examples for the case of NAS and feature map selection that we use for the rest of the paper. Our main observation is that weight-sharing for architecture search extends the hypothesis space to be further parameterized by a finite set of configurations C. Formally, we have a structured hypothesis space H(C, W) = {h w ∈ H(C, W) with low population error w (x), y) for loss: Y × Y → R. Hence, we can apply ERM as usual, with optional regularization, to select a hypothesis from the extended hypothesis space; in fact this is done by some NAS methods (e.g.,). The learning algorithm is then min w∈W,c∈C for block specific regularizers R W and R C. Note that in the absence of weight-sharing, we would need to learn a separate hypothesis h (c) wc for each hypothesis subclass H c. Although a brute force approach to selecting a hypothesis from H(C, W) via ERM would in effect require this as well, our subsequent examples demonstrate how the weight-sharing construct allows us to apply more efficient gradient-based optimization approaches, which we study in Section 3. Feature Map Selection: In this setting, the structure is induced by a set of feature transformations C = {φ i : X → R n for i = 1, . . ., k}, so the hypothesis space is {f w (φ i (·)): w ∈ W, φ i ∈ C} for some W ⊂ R d. Examples of feature map selection problems include tuning kernel hyperparameters for kernel ridge classification and tuning NLP featurization pipelines for text classification. In these cases f w is a linear mapping f w (·) = w, · and W ⊂ R n. Neural Architecture Search: Weight-sharing methods almost exclusively use micro cell-based search spaces for their tractability and additional structure (; . These search spaces can be represented as directed acyclic graphs (DAGs) with a set of ordered nodes N and edges E. Each node x (i) in the DAG is a feature representation and each edge o (i,j) is an operation on the feature of node j passed to node i and aggregated with other inputs to form x (j), with the restriction that a given node j can only receive edges from prior nodes as input. Hence, the feature at a given node i is. Search spaces are then specified by the number of nodes, the number of edges per node, and the set of operations O that can be applied at each edge. In this case the structure C ⊂ {0, 1} |E||O| of the hypothesis space is the set of all valid architectures for this DAG encoded by edge and operation decisions. Treating both weights w ∈ W and architecture decision c ∈ C as parameters, weight-sharing methods train a single shared-weights network h (c) w: X → Y encompassing all possible functions within the search space. Therefore, the sharedweights network includes all possible edges between nodes and all possible operations per edges. In addition to the weights w ∈ W corresponding to all the operations, the shared-weights network also takes architecture weights c ∈ C as input, where c indicates the weight given to operation o on edge (i, j) so that the feature of a given node i is Gradient-based weight-sharing methods apply continuous relaxations to the architecture search space in order to compute gradients. Some methods like DARTS and its variants (; ; ; ; relax the search space by considering a mixture of operations per edge and then discretize to a valid architecture in the search space. With the mixture relaxation, we replace all c ∈ {0, 1} |E||O| in the above expressions by continuous counterparts θ ∈ |E||O|, with the constraint that o∈O θ (i,j) o = 1, i.e., the architecture weights for operations on each edge sum to 1. Other methods like SNAS, ASNG-NAS , and ProxylessNAS assume a parameterized distribution p θ from which architectures are sampled. By substituting continuous parameters θ ∈ Θ in for discrete parameters c ∈ C, we are able to use gradient-based methods to optimize. We address the question of how to effectively use gradient optimization for weight-sharing in the next section. While continuous relaxation enables state-of-the-art , architecture search remains expensive and noisy, with state-of-the-art mixture methods requiring second-order computations and probabilistic methods suffering from high variance policy gradients. Moreover, while the use of SGD to optimize network weights is a well-tested approach, architecture weights typically lie in constrained, non-Euclidean geometries in which other algorithms may be more appropriate. Recognizing this, several efforts have attempted to derive a better optimization schemes; however, the associated guarantees for most of them hold for auxiliary objectives, such as regret of local configuration decisions, that are not connected to the optimization objective and ignore the two-block nature of the problem ). do consider an alternating descent method for the training objective, their are asymptotic and certainly do not indicate any finite-time convergence. In this section we address these difficulties by showing that the mirror descent (MD) framework is the right tool for designing algorithms in the block optimization problems that occur in architecture search. We describe how such geometry-aware gradient algorithms lead to faster stationary-point convergence; as we will show in Section 5, this yields simple, principled, and effective algorithms for large-scale multi-geometry problems such as NAS. Algorithm 1: Two geometry-aware optimization algorithms for multi-block optimization for a β-strongly-smooth function over if SBMD Problem Geometry: We relax optimization problems of the form to the problem of minimizing a function f: X → R over a convex product space X i consisting of blocks i, each with an associated norm · (i). For example, in typical NAS we can set X 1 to be the set-product of |E| simplices over the operations O and the associated norm to be · 1 while X 2 = W ⊂ R d is the space of network weights and the associated norm is · 2. To each block i we further associate a distance-generating function (DGF) ω i: X → R that is 1-strongly-convex w.r.t. . For example, in the Euclidean case using ω 2 (·) = yields the usual squared Euclidean distance; over the probability simplex we often use the entropy ω 1 (·) = ·, log(·), which is 1-strongly-convex w.r.t. · 1 and for which D ω1 is the KL-divergence. Given an unbiased gradient estimate g(x t, ζ) = E ζ ∇f (x t), the (single-block) stochastic MD step is for some learning rate η > 0. In the Euclidean setting this reduces to SGD, while with the entropic regularizer the iteration becomes equivalent to exponentiated gradient. Key to the guarantees for MD is the fact that the dual norm of the problem geometry is used to measure the second moment of the gradient; if every coordinate of g is bounded a.s. by σ, then under Euclidean geometry the dependence is on E g(x) 2 2, * = E g(x) 2 2 ≤ σ 2 d while using entropic regularization it is on. Thus in such constrained 1 geometries mirror descent can yield dimension-free convergence guarantees. While in this paper we focus on the benefit in this simple case, the MD meta-algorithm can be used for many other geometries of interest in architecture search, such as for optimization over positive-definite matrices . Algorithms and Guarantees: We propose two methods for the above multi-block optimization problems: stochastic block mirror descent (SBMD) and alternating successive convex approximation (ASCA). At each step, both schemes pick a random coordinate i to update; SBMD then performs a mirror descent update similar to but with a batched gradient, while ASCA optimizes a strongly-convex surrogate function using a user-specified solver. Note that both methods require that f is β-strongly-smooth w.r.t. each block's norm · (i) to achieve convergence guarantees, a standard assumption in stochastic non-convex optimization . This condition holds for the architecture search under certain limitations, such as a restriction to smooth activations. In the supplement we also show that in the single-block case ASCA converges under the more general relative-weak-convexity criterion . We first discuss SBMD, for which non-convex convergence guarantees were shown by; this algorithm is the one we implement in Section 5 for NAS. A first issue is how to measure stationarity in constrained, non-Euclidean geometries. In the single-block setting we can set a smoothness-dependent constant λ > 0 and measure how far the proximal gradient operator prox∇ λ (x) = arg min u∈X λ ∇f (x), u + D ω (u||x) is from a fixed point, which yields the projected gradient measure. Notably, in the unconstrained Euclidean case this yields the standard stationarity measure ∇f (x) 2 2. For the multi-block case we replace an ε-stationary point of f w.r.t. the projected gradient if G λ (x) ≤ ε. In the non-Euclidean case the norm of the projected gradient measures how far we are from satisfying a first-order optimality condition, namely how far the negative gradient is from being in the normal cone of f to the set X (, Proposition 4.1). For this condition, Algorithm 1 has the following guarantee: Theorem 3.1 . If f is β-strongly-smooth, F = f (x ) − min u∈X f (u), and oracle calls to reach an ε-stationary-point x ∈ X as measured by G 1 We next provide a guarantee for ASCA in the form of a reduction to strongly-convex optimization algorithms. This is accomplished by the construction of the solving of a surrogate function at each iteration, which in the Euclidean case is effectively adding weight-decay. ASCA is useful in the case when efficient strongly-convex solvers are available for some or all of the blocks; for example, this is frequently the case for feature map selection problems such as our kernel approximation examples, which employ 2 -regularized ERM in the inner optimization. Taking inspiration from , for ASCA we analyze a stronger notion of stationarity that upper-bounds G λ (x) 2, which we term the projected stationarity measure:. Note that this is not the same measure used by , although in the appendix we show that in the single-block case our holds for their notion also. Theorem 3.2. If f is β-strongly-smooth and where ε > 0 is the solver tolerance and the expectation is over the randomness of the algorithm and associated oracles. Proof Summary. The proof generalizes a of and is in Appendix A. Thus if we have solvers that return approximate optima of strongly-convex functions on each block then we can converge to a stationary point of the original function. The convergence rate will depend on the solver used; for concreteness we give a specification for stochastic mirror descent. Corollary 3.1. Under the conditions of Theorem 3.1, if on i ASCA uses the Epoch-GD method of ωi oracle calls suffice to reach an ε-stationary-point x ∈ X as measured by E ∆ 1 This oracle complexity matches that of Theorem 3.1 apart from the extra β 2 L 2 ωi term due to the surrogate function, which we show below is in practice a minor term that does not obviate the benefit of geometry-awareness. On the other hand, ASCA is much more general, allowing for many different algorithms on individual blocks; for example, many popular neural optimizers such as Adam have variants with strongly-convex guarantees . The Benefit of Geometry-Aware Optimization: We conclude this section with a formalization of how convergence of gradient-based architecture-search algorithms can benefit from this optimization strategy. Recalling from our example, we have the architecture parameter space X 1 consisting of |E| simplices over |O| variables equipped with the 1-norm and the shared weight space X 2 = W equipped with the Euclidean norm. We suppose that the stochastic gradients along each block i have coordinates bounded a.s. by σ i > 0. Then if we run SBMD using SGD to optimize the shared weights and exponentiated gradient to update the architecture parameters, Theorem 3.1 implies that we reach an ε-stationary point in O stochastic gradient computations. The main benefit here is that the first term in the numerator is not σ 2 1 |E||O|, which would be the case if we used SGD; this improvement is critical as the noise σ 2 1 of the architecture gradient can be very high, especially if a policy gradient is used to estimate probabilistic derivatives. In the case of ASCA, we can get similar guarantees assuming the probabilities for each operation are lower-bounded by some small δ > 0 and that the space of shared weights is bounded by B; then the guarantee will be as above except with an additional O(log 2) term (independent of σ 1, σ 2). While for both SBMD and ASCA the σ 2 2 d term from training the architecture remains, this will be incurred even in single-architecture training using SGD. Furthermore, in the case of ASCA it may be improved using adaptive algorithms . In Section 2, we described the weight-sharing hypothesis class H(C, W) as a set of functions nondisjointly partitioned by a set of configurations C sharing weights in W and posed the ERM problem associated with selecting a hypothesis from H(C, W). However, as mentioned in Section 1, the objective solved in practice is a bilevel problem where a separate validation set is used for architecture parameter updates. Formally, the bilevel optimization problem considered is min w∈W,c∈C where T, V ⊂ Z is a pair of training/validation sets sampled i.i.d. from D, the upper objective w (x), y) is the empirical risk over V, and L T (w, c) is some objective induced by T. We intentionally differentiate the two losses since training is often regularized. This setup is closely related to the well-studied problems of model selection and cross-validation. However, a key difference is that the choice of configuration c ∈ C does not necessarily provide any control over the complexity of the hypothesis space; for example, in NAS as it is often unclear how the hypothesis space changes due to the change in one decision. By contrast, the theory of model selection is often directly concerned with control of model complexity. Indeed, in possibly the most common setting the hypothesis classes are nested according to some total order of increasing complexity, forming a structure . This is for example the case in most norm-based regularization schemes. Even in the non-nested case, there is often an explicit tradeoff between parsimony and accuracy . With the configuration parameters in architecture search behaving more like regular model parameters rather than as controls on the model complexity, it becomes reasonable to wonder why most NAS practitioners have used the bilevel formulation. Does the training-validation split exploit the partitioning of the hypothesis space H(C, W) induced by the configurations C? To see when this might be true, we first note that a key aspect of the optima of the bilevel weight-sharing problem is the restriction on the model weights -that they must be in the set arg min w∈W L T (h (c) w ) of the inner objective L T. As we will see, under certain assumptions this can reduce the complexity of the hypothesis space without harming performance. First, for any sample w )} be the version space (, Equation 6) induced by some configuration c and the objective function. Second, let N (F, ε) be the L ∞ -covering-number of a set of functions F at scale ε > 0, i.e. the number of L ∞ balls required to construct an ε-cover of F (, Equation 3 .60). These two quantities let us define a complexity measure over the shared weight hypothesis space: The version entropy is a data-dependent quantification of how much the hypothesis class is restricted by the inner optimization. For finite C, a naive bound shows that Λ(H, ε, T) is bounded by log |C| + max c∈C log N (H c (T), ε), so that the second term measures the worst-case complexity of the global minimizers of L T. In the feature selection problem, L T is usually a strongly-convex loss due to regularization and so all version spaces are singleton sets, making the version entropy log |C|. In the other extreme case of nested model selection the version entropy reduces to the complexity of the version space of the largest model and so may not be informative. However, in practical problems such as NAS an inductive bias is often imposed via constraints on the number of input edges. To bound the excess risk in terms of the version entropy, we first discuss an important assumption that describes cases when we expect the shared weights approach to perform well: Assumption 4.1. There exists a good c * ∈ C, i.e. one satisfying (w *, c w) for some w * ∈ W, such that w.h.p. over the drawing of training set T ∼ D m T at least one of the minima of the optimization induced by c * and T has low excess risk, i.e. w.p. 1 − δ there exists This assumption requires that w.h.p. the inner optimization objective does not exclude all low-risk classifiers for the optimal configuration. Note that it asks nothing of either the other configurations in C, which may be arbitrarily bad, nor of the hypotheses found by the procedure. It does however prevent the case where one knows the optimal configuration but minimizing the provided objective L T does not provide a set of good weights. Note that if the inner optimization is simply ERM over the training set T, i.e. L T = T, then standard learning-theoretic guarantees will give ε * exc (m T, δ) decreasing in m T and increasing at most poly-logarithmically in 1 δ. With this assumption, we can show the following guarantee on solutions to the bilevel optimization. Theorem 4.1. Letĥ be a hypothesis corresponding to the solution of the bilevel optimization. Then under Assumption 4.1 if is B-bounded we have w.p. 1 − 3δ that The first difference is bounded by the version entropy usng the constraint onĥ ∈ H c, the second by optimality ofĥ on V, the third by Hoeffding's inequality, and the last by Assumption 4.1. As shown in the applications below, the significance of this theorem is that a bound on the version entropy guarantees excess risk almost as good as that of the (unknown) optimal configuration without assuming anything about the complexity or behavior of sub-optimal configurations. Feature Map and Kernel Selection: In the feature map selection problem introduced in Section 2, } is a set of feature maps and the inner problem L T is 2 -regularized ERM for linear classification over the ing feature vectors. The bilevel problem is then Due to strong-convexity of L T, each map φ i induces a unique minimizing weight w ∈ W and thus a singleton version space, therefore upper bounding the version entropy by log |C| = log N. Furthermore, for Lipschitz losses and appropriate choice of regularization coefficient, standard for 2 -regularized ERM for linear classification (e.g. In the special case of kernel selection using random Fourier approximation, we can apply associated generalization guarantees (, Theorem 1) to show that we can compete with the optimal RKHS from among those associated with one of the configurations: Corollary 4.2. In feature map selection suppose each map φ ∈ C is associated with a random Fourier feature approximation of a continuous shift-invariant kernel that approximates an RKHS H φ and is the square loss. If the number of features In both cases we are able to get risk bounds almost identical to the excess risk achievable if we knew the optimal configuration beforehand, up to an additional capacity term depending weakly on the number of configurations. This would not be possible with solving the regular ERM objective instead of the bilevel optimization as we would then have to contend with the possibly high complexity of the hypothesis space induced by the worst configuration. Neural Architecture Search: In the case of NAS we do not have a bound on the version entropy, which now depends on all of C. Whether the version space, and thus the complexity, of deep networks is small compared to the number of samples is unclear, although we gather some evidence. number of critical points is exponential only in the number of layers, which would yield a small version entropy. It is conceivable that the quantity may be further bounded by the complexity of solutions explored by the algorithm when optimizing L T ; indeed, we find that shared-weight optimization leads to models with smaller 2 -norm and distance from initialization than from-scratch SGD on a single network (see Appendix D.4). On the other hand, argue, with evidence in restricted settings, that even the most stringent implicit regularization cannot lead to a non-vacuous uniform convergence bound; if true more generally this would imply that the NAS version entropy is quite large. Here we demonstrate how weight-sharing can be used as a tool to speed up general architecture search problems by applying it to two feature map selection problems. We then validate our optimization analysis with a geometry-aware weight-sharing method to design CNN cells for CIFAR-10. Feature Map Selection: Recall that here our configuration space has k feature maps φ i: X → R n with outputs passed to a linear classifier w ∈ R n, which will be the shared weights. We will approximate the bilevel optimization with the inner minimization over 2 -regularized ERM λ w 2 2 + (x,y)∈T (w, φ i (x), y). Our weight-sharing procedure starts with a vector θ ∈ ∆ N encoding a probability distribution p θ over [N] and proceeds as follows: according to (an estimate of) its validation loss (x,y)∈V (w (t), φ i (x), y). Observe the equivalence to probabilistic NAS: at each step the classifier (shared parameter) is updated using random feature maps (architectures) on the training samples. The distribution over them is then updated using estimated validation performance. We consider two schemes for this update of θ (t): exponentiated gradient using the score-function estimate and successive elimination, where we remove a fraction of the feature maps that perform poorly on validation and reassign their probability among the remainder. may be viewed as a softer version of, with halving also having only one hyperparameter (elimination rate) and not two (learning rate, stopping criterion). The first problem we consider is kernel ridge regression over random Fourier features on CIFAR-10. We consider three configuration decisions: data preprocessing, choice of kernel, and bandwidth parameter. This problem was considered by , except they fixed the Gaussian kernel whereas we also consider Laplacian; however, they also select the regularization parameter λ, which weight-sharing does not handle. We also study logistic regression for IMDB sentiment analysis of Bag-of-n-Gram (BonG) featurizations, a standard NLP baseline . Here there are eight configuration decisions: tokenization method, whether to remove stopwords, whether to lowercase, choice of n, whether to binarize features, type of feature weighting, smoothing parameter, and post-processing. As some choices affect the feature dimension we hash the BonGs into a fixed number of bins . To test the performance of weight-sharing for feature map selection, we randomly sample 64 configurations each for CIFAR-10 and IMDB and examine whether the above schemes converge to the optimal choice. The main comparison method here is thus random search, which runs a full sweep over these samples; by contrast successive halving will need to solve 6 = log 2 64 regression problems, while for exponentiated gradient we perform early stopping after five iterations. Note that weight-sharing can do no better than random search in terms of accuracy because they are picking a configuration from a space that random search sweeps over. The goal is to see if it consistently returns a good configuration much faster. As our in Figures 1 and 2 show, successive halving indeed does almost as well as random search in much less time. While exponentiated gradient usually does not recover a near-optimal solution, it does on average return a configuration in the top 10%. We also note the strong benefit of over-parameterization for IMDB -the n-gram vocabulary has size 4 million so the number of bins on the right is much larger than needed to learn in a singleconfiguration setting. Overall, these experiments show that weight-sharing can also be used as a fast way to obtain signal in regular learning algorithm configuration and not just NAS. NAS on CIFAR-10: Recall from Section 3 that when the architecture space consists of |E| simplices of dimension |O|, the convergence rate of exponentiated gradient descent to a stationary point of the objective function is independent of the dimension of the space, while SGD has linear dependence. This motivates our geometry-aware method called Exponentiated-DARTS (EDARTS). EDARTS modifies first-order DARTS in two ways. First, in lieu of the softmax operation used by DARTS on the architecture weights, we use standard normalization so that the weight of operation o on edge (i, j) is u. Second, in lieu of Adam, we use exponentiated gradient to update the architecture weights: c t = c t−1 exp(−η∇ c V (h wt−1 (c t−1)). While EDARTS resembles XNAS, our justification for using exponentiated gradient comes directly from aligning with the optimization geometry of ERM. Additionally, EDARTS only requires two straightforward modifications of first-order DARTS, while XNAS relies on a wipeout subroutine and granular gradient-clipping for each edge operation on the cell and data instance level. 1 1 Our own XNAS implementation informed by correspondence with the authors did not produce competitive . We still compare to the architecture XNAS reported evaluated by the DARTS training routine in Table 1. We evaluate EDARTS on the task of designing a CNN cell for CIFAR-10. We use the standard search space as introduced in DARTS ) for evaluation we use the same three stage process used by DARTS and random search with weight-sharing , with stage 3 considered the'final' . We provide additional experimental details in Appendix D. Table 1 shows the performance of EDARTS relative to both manually designed and NAS-discovered architectures. EDARTS finds an architecture that achieves competitive performance with manually designed architectures which have nearly an order-of-magnitude more parameters. Additionally, not only does EDARTS achieve significantly lower test error than first-order DARTS, it also outperforms second order DARTS while requiring less compute time, showcasing the benefit of geometry-aware optimization. Finally, EDARTS achieve comparable performance to the reported architecture for state-of-the-art method XNAS when evaluated using the stage 3 training routine of DARTS. Following XNAS, we also perform an extended evaluation of the best architecture found by EDARTS with AutoAugment, cosine power annealing , cross-entropy with label smoothing, and trains for 1500 epochs. We evaluated the XNAS architecture using our implementation for a direct comparison and also to serve as a reproducibility check. EDARTS achieved a test error of 2.18% in the extended evaluation compared to 2.15% for XNAS in our reproduced evaluation; note the published test error for XNAS is 1.81%. To meet a higher bar for reproducibility we report'broad reproducibility' by repeating the entire pipeline from stage 1 to stage 3 for two additional sets of seeds. Our in Table 2 (see Appendix) show that EDARTS has lower variance across experiments than random search with weight sharing . However, we do observe non-negligible variance in the performance of the architecture found by different random seed initializations of the shared-weights network, necessitating running multiple searches before selecting an architecture. A OPTIMIZATION This section contains proofs and generalizations of the non-convex optimization in Section 3. We first gather some necessary definitions and from convex analysis. Definition A.1. Let X be a convex subset of a finite-dimensional real vector space and f: X → R be everywhere sub-differentiable. 1. For α > 0, f is α-strongly-convex w.r.t. norm · if ∀ x, y ∈ X we have Definition A.2. Let X be a convex subset of a finite-dimensional real vector space. The Bregman divergence induced by a strictly convex distance-generating function (DGF) ω: X → R is D ω (x||y) = ω(x) − ω(y) − ∇ω(y), x − y ∀ x, y ∈ X By definition, the Bregman divergence satisfies the following properties: 3. If ω is β-strongly-smooth w.r.t. norm · then so is D ω (·||y) ∀ y ∈ X. Furthermore, D ω (x||y) ≤ β 2 x − y 2 ∀ x, y ∈ X. Lemma A.1 (Three-Points Lemma). (, Lemma 9.11) For any DGF ω: X → R and all x, y, z ∈ X we have ∇ω(y) − ∇ω(x), z − x = D ω (z||x) + D ω (x||y) − D ω (z||y) Definition A.3. Let ω: X → R be a 1-strongly-convex DGF. Then for constant λ > 0 and an everywhere sub-differentiable function f: X → R the proximal operator is defined over x ∈ X as prox λ (x) = arg min Note that the prox operator is well-defined whenever f is β-strongly-smooth for some β < λ. We will also use the following notation for the proximal gradient operator: Note that the prox grad operator is always well-defined. Theorem A.1. (, Theorem 9.12) For any λ > 0, 1-strongly-convex DGF ω: X → R, and x ∈ X let f: X → R be an everywhere sub-differentiable function s.t. λf (·) + D ω (·||x) is convex over X. Then for x + = prox λ (x) and all u ∈ X we have Lemma A.2. For any λ > 0, 1-strongly-convex DGF ω: X → R, and x ∈ X let f: X → R be an everywhere sub-differentiable function s.t. Proof. Applying Theorem A.1 followed by Lemma A.1 yields Corollary A.1. For any λ > 0, 1-strongly-convex DGF ω: X → R, x ∈ X, and everywhere sub-differentiable function f: X → R we have for Because we consider constrained non-convex optimization, we cannot measure convergence to a stationary point by the norm of the gradient. Instead, we analyze the convergence proximal-mappingbased stationarity measures. The most well-known measure is the norm of the projected gradient , which in the unconstrained Euclidean case reduces to the norm of the gradient and in the general case measure the distance between the current point and a cone satisfying first-order optimality conditions (, Proposition 4.1). Our convergence hold for a stronger measure that we call the projected stationarity and which is inspired by the Bregman stationarity measure of but using the prox grad operator instead of the prox operator. Definition A.4. Let ω: X → R be a 1-strongly-convex DGF and f: X → R be an everywhere sub-differentiable function. Then for any λ > 0 we define the following two quantities: The following properties follow: We can also consider the Bregman stationarity measure of directly. As this measure depends on the prox operator, which is not always defined, we first state the notion of non-convexity that consider. Definition A.5. An everywhere sub-differentiable function f: X → R is (γ, ω)-relatively-weaklyconvex ((γ, ω)-RWC) for γ > 0 and ω: X → R a 1-strongly-convex DGF if f (·) + γω(·) is convex over X. Note that all γ-strongly-smooth functions are (γ, ω)-RWC. Note that (γ, ω)-RWC is a generalization of γ-strong-smoothness w.r.t. the norm w.r.t. which ω is strongly-convex. Furthermore, for such functions we can always define the prox operator for λ > γ, allowing us to also define the Bregman gradient below. Similarly to before, bounding the Bregman stationarity measure yields a stronger notion of convergence than the squared norm of the Bregman gradient. For the relationship between the Bregman stationarity measure and first-order optimality conditions see Zhang and He (2018, Equation 2.11). Definition A.6. Let ω: X → R be a 1-strongly-convex DGF and f: X → R be a (γ, ω)-RWC everywhere sub-differentiable function for some γ > 0. Then for any λ > γ we define the following two quantities: Here we prove our main optimization . We begin with a descent lemma guaranteeing improvement of a non-convex function due to approximately optimizing a strongly convex surrogate. Lemma A.3. Let ω: X → R be a 1-strongly-convex DGF and f: X → R be everywhere subdifferentiable. For some x ∈ X and ρ > 0 definef x (·) = f (·) + ρD ω (·||x) and letx ∈ X be a point s.t. Ef x (x) − min u∈Xfx (u) ≤ ε. Then 1. If f is β-strongly-smooth, ρ > β, and λ = Proof. Generalizing an argument in Agarwal et al. (2019, Theorem A.2), for x + ∈ X we have by strong-convexity off x that If f is β-strongly-smooth set x + = prox∇ λ (x), so that by Corollary A.1 we have In the other case of f being (γ, ω)-RWC set x + = prox λ (x), so that by Lemma A.2 we have We now turn to formalizing our multi-block setting and assumptions. Setting A.1. For i = 1,..., b let X i be a convex subset of a real vector space with an associated DGF ω i: X i → R that is 1-strongly-convex w.r.t. some norm · (i) over X i. We have an everywhere sub-differentiable function f: X → R over the product space Our main will hold for the case when the following general assumption is satisfied. We will later show how this assumption can follow from strong smoothness or relative weak convexity and existing algorithmic . Assumption A.1. In Setting A.1, for any given ε > 0 and each i ∈ [b] there exists a constant ρ i > 0 and an algorithm A i: X → X that takes a point x ∈ X and returns a pointx ∈ X satisfyinĝ x −i = x −i and where the subscript i selects block i, the subscript −i selects all blocks other than block i, and E Ai denotes expectation w.r.t. the randomness of algorithm A i and any associated stochastic oracles. Algorithm 2: Generic successive convex approximation algorithm for reaching a stationary point of the non-convex function in Setting A.1. Input: Point x ∈ X in the product space of Setting A.1. Algorithms A 1,..., Our main relies on the following simple lemma guaranteeing non-convex convergence for a generic measure satisfying guaranteed expected descent: Lemma A.4. In Setting A.1, for some ε > 0 and each λi (x) be any measure s.t. for some λ i and some algorithm A i: X → X we have Then the output x of Algorithm 2 satisfies where F = f (x ) − arg min u∈X f (x) and the expectation is taken over the sampling at each iteration, the sampling of the output, and the randomness of the algorithms and any associated stochastic oracles. Proof. Define Ξ t = {(ξ s, A ξs)} t s=1 and note that x (t+1) = A ξt (x (t) ). We then have In the single-block setting, Lemmas A.3 and A.4 directly imply the following guarantee: Theorem A.2. In Setting A.1 and under Assumption A.1, let b = 1 and ρ satisfy one of the following: 1. f: X → R is β-strongly-smooth and ρ = 2β. 2. f: X → R is (γ, ω)-RWC for some DGF ω: X → R and ρ = 2γ. Then Algorithm 2 returns a point x ∈ X satisfying one of the following (respectively, w.r.t. the above settings) for Here the expectation is taken over the randomness of the algorithm and oracle. We can apply a known for the strongly convex case to recover the rate of for non-convex stochastic mirror descent, up to an additional depending on ω: Corollary A.2. In Setting A.1 for b = 1 and (γ, ω)-RWC f, suppose we have access to f through a stochastic gradient oracle g(x) = E∇f (x) such that E g 2 * ≤ G 2. Let A: X → X be an algorithm that for any x ∈ X runs the Epoch-GD method of with total number of steps N, initial epoch length T 1 = 4 and initial learning rate η 1 = 1 γ onf x (·) = f (·) + 2γD ω (·||x). Then with N T calls to the stochastic gradient oracle Algorithm 2 returns a point x ∈ X satisfying and L ω the Lipschitz constant of ω w.r.t. · over X. So an expected ε-stationary-point, as measured by Proof. Apply Theorem 5 of together with the fact thatf x is γ-stronglyconvex w.r.t. · and its stochastic gradient is bounded by For the multi-block case our hold only the projected stationarity measure: Theorem A.3. In Setting A.1 and under Assumption A.1 assume f (·, x −i) is β-strongly-smooth w.r.t. Here the expectation is taken over the randomness of the algorithm and oracle and the projected stationarity measure ∆ λ is defined w.r.t. the Bregman divergence of the DGF ω(where prox∇ To apply Lemma A.4 in the multiblock setting it suffices to show that the sum of the projected stationarity measures on each block is equal to the projected stationarity measure induced by the sum of the DGFs. For some λ > 0 and any i ∈ [b] we have that and so Thus applying Lemma A.2 with λ = 1 4ρ yields the . In the following corollary we recover the rate of for non-convex blockstochastic mirror descent, up to an additional term depending on ω i: Corollary A.3. In Setting A.1 for β-strongly-smooth f, suppose we have access to f through a stochastic gradient oracle g(x) = E∇f (x) such that E g i i. For i ∈ [b] let A i: X → X be an algorithm that for any x ∈ X runs the Epoch-GD method of with total number of steps N, initial epoch length T 1 = 4 and initial learning rate η 1 = 1 γ on surrogate functionf x (·) = f (·, x −i) + 2βD ωi (·||x i). Then with N T calls to the stochastic gradient oracle Algorithm 2 returns a point x ∈ X satisfying We can specialize this to the architecture search setting where we have a configuration search space contained in the product of simplices induced by having n decisions with c choices each together with a parameter space bounded in Euclidean norm. Corollary A.4. Under the assumptions of Corollary A.3, suppose b = 2 and we have the following two geometries: Suppose the stochastic gradient oracle of f has bounded ∞ -norm σ 1 over X 1 and σ 2 over X 2. Then Algorithm 2 will return an expected ε-stationary point of f under the projected stationarity measure in a number of stochastic oracle calls bounded by This section contains proofs of the generalization in Section 4. We first describe the setting for which we prove our general . Setting B.1. Let C be a set of possible architecture/configurations of finite size such that each c ∈ C is associated with a parameterized hypothesis class H c ={h (c) w: X → Y: w ∈ W} for input space Z = X × Y and fixed set of possible weights W. We will measure the performance of a hypothesis h (c) w on an input z = (x, y) ∈ Z using z (w, c) = (h Finally, we will consider solutions of optimization problems that depend on the training data and architecture. Specifically, for any configuration c ∈ C and finite subset S ⊂ Z let W c (S) ⊂ W be the set of global minima of some optimization problem induced by S and c and let the associated version space be H c (S) = {h w : X → Y is determined by a choice of architecture c ∈ C and a set of network weights w ∈ W and the loss : Y × Y → {0, 1} is the zero-one loss. In the simplest case W c (S) is the set of global minima of the ERM problem min We now state the main assumption we require. Assumption B.1. In Setting B.1 there exists a good architecture c * ∈ C, i.e. one satisfying (w *, c *) ∈ arg min W×C D (w, c) for some weights w * ∈ W, such that w.p. 1 − δ over the drawing of training set T ∼ D m T at least one of the minima of the optimization problem induced by c * and T has low excess risk, i.e. ∃ w ∈ W c * (T) s.t. for some error function ε c *. Clearly, we prefer error functions ε c * that are decreasing in the number of training samples m T and increasing at most poly-logarithmically in 1 δ. This assumption requires that if we knew the optimal configuration a priori, then the provided optimization problem will find a good set of weights for it. We will show how, under reasonable assumptions, Assumption B.1 can be formally shown to hold in Settings B.2 and B.3. Our general will be stated in terms of covering numbers of certain function classes. Definition B.1. Let H be a class of functions from X to Y. For any ε > 0 the associated L ∞ covering number N (H, ε) of H is the minimal positive integer k such that H can be covered by k balls of L ∞ -radius ε. The following is then a standard in statistical learning theory (see e.g. Lafferty et al. (2010, Theorem 7.82 where we use the loss notation from Setting B.1. Before stating our theorem, we define a final quantity, which measures the log covering number of the version spaces induced by the optimization procedure over a given training set. Definition B.2. In Setting B.1, for any sample S ⊂ X × Y define the version entropy to be Λ(H, ε, S) = log N c∈C H c (S), ε. Theorem B.2. In Setting B.1 let (ŵ,ĉ) ∈ W × C be obtained as a solution to the following optimization problem: arg min Then under Assumption B.1 we have w.p. 1 − 3δ that each term of which can be bounded as follows: 1. Sinceŵ ∈ Wĉ(T) for someĉ ∈ C the hypothesis space can be covered by the union of the coverings of H c (T) over c ∈ C, so by Theorem B.1 we have that w. 2. By optimality of the pair (ŵ,ĉ) and the fact that w ∈ W c * (T) we have 3. Hoeffding's inequality yields V (w, c We can then directly apply Theorem B.2 and the fact that the version entropy is bounded by log |C| because the minimizer over the training set is always unique to get the following: Corollary B.2. In Setting B.2 let (ŵ,ĉ) ∈ W × C be obtained as a solution to the following optimization problem: arg min In the special case of kernel selection we can apply generalization for learning with random features to show that we can compete with the optimal RKHS from among those associated with one of the configurations (, Theorem 1): Corollary B.3. In Setting B.2, suppose each configuration c ∈ C is associated with a random Fourier feature approximation of a continuous shift-invariant kernel that approximates an RKHS H c. Suppose is the squared loss so that (ŵ,ĉ) ∈ W × C is obtained as a solution to the following optimization problem: In the case of neural architecture search we are often solving (unregularized) ERM in the inner optimization problem. In this case we can make an assumption weaker than Assumption B.1, namely that the set of empirical risk minimizers contains a solution that, rather than having low excess risk, simply has low generalization error; then applying Hoeffding's inequality yields the following: Corollary B.4. In Setting B.1 let (ŵ,ĉ) ∈ W × C be obtained as a solution to the following optimization problem: arg min Suppose there exists c * ∈ C satisfying (w *, c *) ∈ arg min W×C D (w, c) for some weights w * ∈ W such that w.p. 1 − δ over the drawing of training set T ∼ D m T at least one of the minima of the optimization problem induced by c * and T has low generalization error, i.e. ∃ w ∈ arg min w ∈W T (w, c *) s.t. Solvers for Ridge regression and logistic regression were from scikit-learn . For CIFAR-10 we use the kernel configuration setting from but replacing the regularization parameter by the option to use the Laplace kernel instead of Gaussian. The regularization was fixed to λ = 1 2 The split is 40K/10K/10K. For IMDB we consider the following configuration choices: For stages 2 and 3, we train each architecture for 600 epochs with the same hyperparameter settings as DARTS. For completeness, we describe the convolutional neural network search space considered. The set of operations O considered at each node include: 3 × 3 separable convolution, 5 × 5 separable convolution, 3×3 dilated convolution, 5×5 dilated convolution, max pooling, average pooling, identity. We use the same search space to design a "normal" cell and a "reduction" cell; the normal cells have stride 1 operations that do not change the dimension of the input, while the reduction cells have stride 2 operations that half the length and width dimensions of the input. In the experiments, for both cell types, we set N = 6 with 2 input nodes and 4 intermediate nodes, after which the output of all intermediate nodes are concatenated to form the output of the cell. We use EDARTS to train a smaller shared-weights network in the search phase with 8 layers and 24 initial channels instead of the 16 used by DARTS. Additionally, to more closely mirror the architecture used for evaluation in stage 2 and 3, we use an auxiliary head with weight 0.4 and scheduled path dropout of 0.2. For the EDARTS architecture updates, we use a learning rate of 0.2 for the normal cell and 0.6 for the reduction cell. All other hyperparameters are the same as DARTS: 50 training epochs, batch size of 64, gradient clipping of 5 for network weights, SGD with momentum set to 0.9 and learning rate annealed from 0.025 to 0.001 with cosine annealing , and weight decay of 0.0003. We use the same evaluation scheme as DARTS when retraining architectures from scratch. The larger evaluation network has 20 layers and 36 initial channels and is trained for 600 epochs using SGD with momentum set to 0.9, a batch size of 96, and a learning rate of 0.025 annealed down to 0; the gradient clipping scheduled drop path rate and weight decay are identical to the search phase. We also use an auxiliary head with a weight of 0.4 and cutout . We investigate whether weight-sharing implicitly regularizes the hypothesis space by examining the 2 norms and distance from initialization of the shared-weights network relative to that observed when training the best EDARTS architecture from scratch. We use the same network depth and hyperparameters as those used for the shared-weights network to train the fixed architecture. Figure 4 shows the percent difference in the norms between the fixed architecture and the shared-weights network pruned to just the operations kept for the fixed architecture. From the chart, we can see that both the 2 distance from initialization and the 2 norm of the shared-weights is smaller than that of a fixed network are higher than that of the shared-weights network by over 40%, suggesting weight-sharing acts as a form of implicit regularization. Table 2 show that EDARTS has lower variance across experiments than random search with weight sharing . However, we do observe non-negligible variance in the performance of the architecture found by different random seed initializations of the shared-weights network, necessitating running multiple searches before selecting an architecture.' it should be close to 'An analysis of the learning and optimization structures of architecture search in neural networks and beyond.' but not to 'We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.'.

Question: Coding theory is a central discipline underpinning wireline and wireless modems that are the workhorses of the information age. Progress in coding theory is largely driven by individual human ingenuity with sporadic breakthroughs over the past century. In this paper we study whether it is possible to automate the discovery of decoding algorithms via deep learning. We study a family of sequential codes parametrized by recurrent neural network (RNN) architectures. We show that cre- atively designed and trained RNN architectures can decode well known sequential codes such as the convolutional and turbo codes with close to optimal performance on the additive white Gaussian noise (AWGN) channel, which itself is achieved by breakthrough algorithms of our times (Viterbi and BCJR decoders, representing dynamic programing and forward-backward algorithms). We show strong gen- eralizations, i.e., we train at a specific signal to noise ratio and block length but test at a wide range of these quantities, as well as robustness and adaptivity to deviations from the AWGN setting. Reliable digital communication, both wireline (ethernet, cable and DSL modems) and wireless (cellular, satellite, deep space), is a primary workhorse of the modern information age. A critical aspect of reliable communication involves the design of codes that allow transmissions to be robustly (and computationally efficiently) decoded under noisy conditions. This is the discipline of coding theory; over the past century and especially the past 70 years (since the birth of information theory BID22) much progress has been made in the design of near optimal codes. Landmark codes include convolutional codes, turbo codes, low density parity check (LDPC) codes and, recently, polar codes. The impact on humanity is enormous -every cellular phone designed uses one of these codes, which feature in global cellular standards ranging from the 2nd generation to the 5th generation respectively, and are text book material BID16.The canonical setting is one of point-to-point reliable communication over the additive white Gaussian noise (AWGN) channel and performance of a code in this setting is its gold standard. The AWGN channel fits much of wireline and wireless communications although the front end of the receiver may have to be specifically designed before being processed by the decoder (example: intersymbol equalization in cable modems, beamforming and sphere decoding in multiple antenna wireless systems); again this is text book material BID26. There are two long term goals in coding theory: (a) design of new, computationally efficient, codes that improve the state of the art (probability of correct reception) over the AWGN setting. Since the current codes already operate close to the information theoretic "Shannon limit", the emphasis is on robustness and adaptability to deviations from the AWGN settings (a list of channel models motivated by practical settings, (such as urban, pedestrian, vehicular) in the recent 5th generation cellular standard is available in Annex B of 3GPP TS 36.101.) (b) design of new codes for multi-terminal (i.e., beyond point-to-point) settings -examples include the feedback channel, the relay channel and the interference channel. Progress over these long term goals has generally been driven by individual human ingenuity and, befittingly, is sporadic. For instance, the time duration between convolutional codes (2nd generation cellular standards) to polar codes (5th generation cellular standards) is over 4 decades. Deep learning is fast emerging as capable of learning sophisticated algorithms from observed data (input, action, output) alone and has been remarkably successful in a large variety of human endeavors (ranging from language BID11 to vision BID17 to playing Go BID23). Motivated by these successes, we envision that deep learning methods can play a crucial role in solving both the aforementioned goals of coding theory. While the learning framework is clear and there is virtually unlimited training data available, there are two main challenges: (a) The space of codes is very vast and the sizes astronomical; for instance a rate 1/2 code over 100 information bits involves designing 2 100 codewords in a 200 dimensional space. Computationally efficient encoding and decoding procedures are a must, apart from high reliability over the AWGN channel. (b) Generalization is highly desirable across block lengths and data rate that each work very well over a wide range of channel signal to noise ratios (SNR). In other words, one is looking to design a family of codes (parametrized by data rate and number of information bits) and their performance is evaluated over a range of channel SNRs. For example, it is shown that when a neural decoder is exposed to nearly 90% of the codewords of a rate 1/2 polar code over 8 information bits, its performance on the unseen codewords is poor. In part due to these challenges, recent deep learning works on decoding known codes using data-driven neural decoders have been limited to short or moderate block lengths BID4 BID13. Other deep learning works on coding theory focus on decoding known codes by training a neural decoder that is initialized with the existing decoding algorithm but is more general than the existing algorithm BID12 BID29. The main challenge is to restrict oneself to a class of codes that neural networks can naturally encode and decode. In this paper, we restrict ourselves to a class of sequential encoding and decoding schemes, of which convolutional and turbo codes are part of. These sequential coding schemes naturally meld with the family of recurrent neural network (RNN) architectures, which have recently seen large success in a wide variety of time-series tasks. The ancillary advantage of sequential schemes is that arbitrarily long information bits can be encoded and also at a large variety of coding rates. Working within sequential codes parametrized by RNN architectures, we make the following contributions. Focusing on convolutional codes we aim to decode them on the AWGN channel using RNN architectures. Efficient optimal decoding of convolutional codes has represented historically fundamental progress in the broad arena of algorithms; optimal bit error decoding is achieved by the'Viterbi decoder' BID27 which is simply dynamic programming or Dijkstra's algorithm on a specific graph (the 'trellis') induced by the convolutional code. Optimal block error decoding is the BCJR decoder BID0 which is part of a family of forward-backward algorithms. While early work had shown that vanilla-RNNs are capable in principle of emulating both Viterbi and BCJR decoders BID28 BID21 we show empirically, through a careful construction of RNN architectures and training methodology, that neural network decoding is possible at very near optimal performances (both bit error rate (BER) and block error rate (BLER)). The key point is that we train a RNN decoder at a specific SNR and over short information bit lengths (100 bits) and show strong generalization capabilities by testing over a wide range of SNR and block lengths (up to 10,000 bits). The specific training SNR is closely related to the Shannon limit of the AWGN channel at the rate of the code and provides strong information theoretic collateral to our empirical . Turbo codes are naturally built on top of convolutional codes, both in terms of encoding and decoding. A natural generalization of our RNN convolutional decoders allow us to decode turbo codes at BER comparable to, and at certain regimes, even better than state of the art turbo decoders on the AWGN channel. That data driven, SGD-learnt, RNN architectures can decode comparably is fairly remarkable since turbo codes already operate near the Shannon limit of reliable communication over the AWGN channel. We show the afore-described neural network decoders for both convolutional and turbo codes are robust to variations to the AWGN channel model. We consider a problem of contemporary interest: communication over a "bursty" AWGN channel (where a small fraction of noise has much higher variance than usual) which models inter-cell interference in OFDM cellular systems (used in 4G and 5G cellular standards) or co-channel radar interference. We demonstrate empirically the neural network architectures can adapt to such variations and beat state of the art heuristics comfortably (despite evidence elsewhere that neural network are sensitive to models they are trained on BID24). Via an innovative local perturbation analysis (akin to BID15)), we demonstrate the neural network to have learnt sophisticated preprocessing heuristics in engineering of real world systems BID10. Among diverse families of coding scheme available in the literature, sequential coding schemes are particularly attractive. They (a) are used extensively in mobile telephone standards including satellite communications, 3G, 4G, and LTE; (b) provably achieve performance close to the information theoretic limit; and (c) have a natural recurrent structure that is aligned with an established family of deep models, namely recurrent neural networks. We consider the basic sequential code known as convolutional codes, and provide a neural decoder that can be trained to achieve the optimal classification accuracy. A standard example of a convolutional code is the rate-1/2 Recursive Systematic Convolutional (RSC) code. The encoder performs a forward pass on a recurrent network shown in FIG0 on binary input sequence b = (b 1, . . ., b K), which we call message bits, with binary vector states (s 1, . . ., s K) and binary vector outputs (c 1, . . ., c K), which we call transmitted bits or a codeword. At time k with binary input b k ∈ {0, 1} and the state of a two-dimensional binary vector s k = (s k1, s k2), the output is a two-dimensional binary vector DISPLAYFORM0 2, where x ⊕ y = |x − y|. The state of the next cell is updated as DISPLAYFORM1 Initial state is assumed to be 0, i.e., s 1 =. The 2K output bits are sent over a noisy channel, with the canonical one being the AWGN channel: the received binary vectors y = (y 1, . . ., y K), which are called the received bits, are y ki = c ki + z ki for all k ∈ [K] and i ∈ {1, 2}, where z ki's are i.i.d. Gaussian with zero mean and variance σ 2. Decoding a received signal y refers to (attempting to) finding the maximum a posteriori (MAP) estimate. Due to the simple recurrent structure, efficient iterative schemes are available for finding the MAP estimate for convolutional codes (vit; BID0 . There are two MAP decoders depending on the error criterion in evaluating the performance: bit error rate (BER) or block error rate (BLER).BLER counts the fraction of blocks that are wrongly decoded (assuming many such length-K blocks have been transmitted), and matching optimal MAP estimator isb = arg max b Pr(b|y). Using dynamic programming, one can find the optimal MAP estimate in time linear in the block length K, which is called the Viterbi algorithm. BER counts the fraction of bits that are wrong, and matching optimal MAP estimator isb k = arg max b k Pr(b k |y), for all k = 1, · · ·, K. Again using dynamic programming, the optimal estimate can be computed in O(K) time, which is called the BCJR algorithm. In both cases, the linear time optimal decoder crucially depends on the recurrent structure of the encoder. This structure can be represented as a hidden Markov chain (HMM), and both decoders are special cases of general efficient methods to solve inference problems on HMM using the principle of dynamic programming (e.g. belief propagation). These methods efficiently compute the exact posterior distributions in two passes through the network: the forward pass and the backward pass. Our first aim is to train a (recurrent) neural network from samples, without explicitly specifying the underlying probabilistic model, and still recover the accuracy of the matching optimal decoders. At a high level, we want to prove by a constructive example that highly engineered dynamic programming can be matched by a neural network which only has access to the samples. The challenge lies in finding the right architecture and showing the right training examples. Neural decoder for convolutional codes. We treat the decoding problem as a K-dimensional binary classification problem for each of the message bits b k. The input to the decoder is a length-2K sequence of received bits y ∈ R 2K each associated with its length-K sequence of "true classes" b ∈ {0, 1} K. The goal is to train a model to find an accurate sequence-to-sequence classifier. The input y is a noisy version of the class b according to the rate-1/2 RSC code defined in earlier in this section. We generate N training examples (DISPLAYFORM2 according to this joint distribution to train our model. We introduce a novel neural decoder for rate-1/2 RSC codes, we call N-RSC. It is two layers of bi-direction Gated Recurrent Units (bi-GRU) each followed by batch normalization units, and the output layer is a single fully connected sigmoid unit. Let W denote all the parameters in the model whose dimensions are shown in FIG1, and f W (y) ∈ K denote the output sequence. The k-th output f W (y) k estimates the posterior probability Pr(b k = 1|y), and we train the weights W to minimize the L w error with respect to a choice of a loss function (·, ·) specified below: DISPLAYFORM3 As the encoder is a recurrent network, it is critical that we use recurrent neural networks as a building block. Among several options of designing RNNs, we make three specific choices that are crucial in achieving the target accuracy: (i) bidirectional GRU as a building block instead of unidirectional GRU; (ii) 2-layer architecture instead of a single layer; and (iii) using batch normalization. As we show in Table 1 in Appendix C, unidirectional GRU fails because the underlying dynamic program requires bi-directional recursion of both forward pass and backward pass through the received sequence. A single layer bi-GRU fails to give the desired performance, and two layers is sufficient. We show how the accuracy depends on the number of layer in Table 1 in Appendix C. Batch normalization is also critical in achieving the target accuracy. Training. We propose two novel training techniques that improve accuracy of the trained model significantly. First, we propose a novel loss function guided by the efficient dynamic programming, that significantly reduces the number of training example we need to show. A natural L 2 loss (which gives better accuracy than cross-entropy in our problem) would be DISPLAYFORM4 DISPLAYFORM5 Recall that the neural network estimates the posterior Pr(b k = 1|y (i) ), and the true label b DISPLAYFORM6 k is a mere surrogate for the posterior, as typically the posterior distribution is simply not accessible. However, for decoding RSC codes, there exists efficient dynamic programming that can compute the posterior distribution exactly. This can significantly improve sample complexity of our training, as we are directly providing Pr(b k = 1|y (i) ) as opposed to a sample from this distribution, which is b DISPLAYFORM7 k. We use a python implementation of BCJR in to compute the posterior distribution exactly, and minimize the loss DISPLAYFORM8 Next, we provide a guideline for choosing the training examples that improve the accuracy. As it is natural to sample the training data and test data from the same distribution, one might use the same noise level for testing and training. However, this is not reliable as shown in FIG2. Channel noise is measured by Signal-to-Noise Ratio (SNR) defined as −10 log 10 σ 2 where σ 2 is the variance of the Gaussian noise in the channel. For rate-1/2 RSC code, we propose using training data with noise level SNR train = min{SNR test, 0}. Namely, we propose using training SNR matched to test SNR if test SNR is below 0dB, and otherwise fix training SNR at 0dB independent of the test SNR. In Appendix D, we give a general formula for general rate-r codes, and provide an information theoretic justification and empirical evidences showing that this is near optimal choice of training data. Performance. In FIG3, for various test SNR, we train our N-RSC on randomly generated training data for rate-1/2 RSC code of block length 100 over AWGN channel with proposed training SNR of min{SNR test, 1}. We trained the decoder with Adam optimizer with learning rate 1e-3, batch size 200, and total number of examples is 12,000, and we use clip norm. On the left we show bit-errorrate when tested with length 100 RSC encoder, matching the training data. 1 We show that N-RSC is able to learn to decode and achieve the optimal performance of the optimal dynamic programming (MAP decoder) almost everywhere. Perhaps surprisingly, we show on the right figure that we can use the neural decoder trained on length 100 codes, and apply it directly to codes of length 10, 000 and still meet the optimal performance. Note that we only give 12, 000 training examples, while the number of unique codewords is 2 10,000. This shows that the proposed neural decoder (a) can generalize to unseen codeword; and (b) seamlessly generalizes to significantly longer block lengths. More experimental including other types of convolutional codes are provided in Appendix A.We also note that training with b (i) k in decoding convolutional codes also gives the same final BER performance as training with the posterior Pr(b k = 1|y (i) ).Complexity. When it comes to an implementation of a decoding algorithm, another important metric in evaluating the performance of a decoder is complexity. In this paper our comparison metrics focus on the BER performance; the main claim in this paper is that there is an alternative decoding methodology which has been hitherto unexplored and to point out that this methodology can yield excellent BER performance. Regarding the circuit complexity, we note that in computer vision, there have been many recent ideas to make large neural networks practically implementable in a cell phone. For example, the idea of distilling the knowledge in a large network to a smaller network and the idea of binarization of weights and data in order to do away with complex multiplication operations have made it possible to implement inference on much larger neural networks than the one in this paper in a smartphone BID7 BID8. Such ideas can be utilized in our problem to reduce the complexity as well. A serious and careful circuit implementation complexity optimization and comparison is significantly complicated and is beyond the scope of a single paper. Having said this, a preliminary comparison is as follows. The complexity of all decoders (Viterbi, BCJR, neural decoder) is linear in the number of information bits (block length). The number of multiplications is quadratic in the dimension of hidden states of GRU FORMULA8 for the proposed neural decoder, and the number of encoder states for Viterbi and BCJR algorithms. Turbo codes are naturally built out of convolutional codes (both encoder and decoder) and represent some of the most successful codes for the AWGN channel BID1. A corresponding stacking of multiple layers of the convolutional neural decoders leads to a natural neural turbo decoder which we show to match (and in some regimes even beat) the performance of standard state of the art turbo decoders on the AWGN channel; these details are available in Appendix B. Unlike the convolutional codes, the state of the art (message-passing) decoders for turbo codes are not the corresponding MAP decoders, so there is no contradiction in that our neural decoder would beat the message-passing ones. The training and architectural choices are similar in spirit to those of the convolutional code and are explored in detail in Appendix B. In the previous sections, we demonstrated that the neural decoder can perform as well as the turbo decoder. In practice, there are a wide variety of channel models that are suited for differing applications. Therefore, we test our neural decoder under some canonical channel models to see how robust and adaptive they are. Robustness refers to the ability of a decoder trained for a particular channel model to work well on a differing channel model without re-training. Adaptivity refers to the ability of the learning algorithm to adapt and retrain for differing channel models. In this section, we demonstrate that the neural turbo decoder is both adaptive and robust by testing on a set of non-Gaussian channel models. Robustness. The robustness test is interesting from two directions, other than obvious practical value. Firstly, it is known from information theory that Gaussian noise is the worst case noise among all noise distributions with a given variance BID22 BID9. Shannon showed in his original paper BID22 ) that among all memoryless noise sequences (with the same average energy), Gaussian noise is the worst in terms of capacity. After a long time, BID9 showed that for any finite block length, the BER achieved by the minimum distance decoder for any noise pdf is lower bounded by the BER for Gaussian noise under the assumption of Gaussian codebook. Since Viterbi decoding is the minimum distance decoder for convolutional codes, it is naturally robust in the precise sense above. On the other hand, turbo decoder does not inherit this property, making it vulnerable to adversarial attacks. We show that the neural decoder is more robust to a non-Gaussian noise, namely, t-distributed noise, than turbo decoder. Secondly, the robust test poses an interesting challenge for neural decoders since deep neural networks are known to misclassify when tested against small adversarial perturbations BID24 BID5 ). While we are not necessarily interested in adversarial perturbations to the input in this paper, it is important for the learning algorithm to be robust against differing noise distributions. We leave research on the robustness to small adversarial perturbations as a future work. For the non-Gaussian channel, we choose the t-distribution family parameterized by parameter ν. We test the performance of both the neural and turbo decoder in this experiment when ν = 3 in Figure 5a and observe that the neural decoder performs significantly better than the standard Turbo decoder (also see FIG0 in Appendix E). In order to understand the reason for such a bad performance of the standard Turbo decoder, we plot the average output log-likelihood ratio (LLR) log p(b k = 1)−log p(b k = 1) as a function of the bit position in Figure 5b, when the input is all-zero codeword. The main issue for the standard decoder is that the LLRs are not calculated accurately (see FIG0 in Appendix E): the LLR is exaggerated in the t-distribution while there is some exaggeration in the neural decoder as well, it is more modest in its prediction leading to more contained error propagation. Adaptivity. A great advantage of neural channel decoder is that the neural network can learn a decoding algorithm even if the channel does not yield to a clean mathematical analysis. Consider a scenario where the transmitted signal is added with a Gaussian noise always, however, with a small probability, a further high variance noise is added. The channel model is mathematically described as follows, with y i describing the received symbol and x i denoting the transmitted symbol at time instant i: DISPLAYFORM0, and w i ∼ N (0, σ 2 b) with probability ρ and w i = 0 with probability 1 − ρ, i.e., z i denotes the Gaussian noise whereas w i denotes the bursty noise. This channel model accurately describes how radar signals (which are bursty) can create an interference for LTE in next generation wireless systems. This model has attracted attention in communications systems community due to its practical relevance BID19 BID20. Under the aforesaid channel model, it turns out that standard Turbo coding decoder fails very badly . The reason that the Turbo decoder cannot be modified in a straight-forward way is that the location of the bursty noise is a latent variable that needs to be jointly decoded along with the message bits. In order to combat this particular noise model, we fine-tune our neural decoder on this noise model, initialized from the AWGN neural decoder, and term it the bursty neural decoder. There are two state-of-the-art heuristics BID18: (a) erasure-thresholding: all LLR above a threshold are set to 0 (b) saturation-thresholding: all LLR above a threshold are set to the (signed) threshold. We demonstrate the performance of our AWGN neural decoder (trained on Gaussian noise) as well as standard turbo decoder (for Gaussian noise) on this problem, shown in Figure 6 when σ b = 3.5, 2, 5. We summarize the of Figure 6: FORMULA3 Interpreting the Neural Decoder We try to interpret the action of the neural decoder trained under bursty noise. To do so, we look at the following simplified model, where y i = x i + z i + w i where x i, y i, z i are as before, but w i = B during the 50-th symbol in a 100-length codeword. We also fix the input codeword to be the all-zero codeword. We look at the average output LLR as a function of position for the one round of the neural decoder in FIG6 and one round of BCJR algorithm in FIG6 (the BER as a function of position is shown in FIG0 in Appendix E). A negative LLR implies correct decoding at this level and a positive LLR implies incorrect decoding. It is evident that both RNN and BCJR algorithms make errors concentrated around the mid-point of the codeword. However, what is different between the two figures is that the scale of likelihoods of the two figures are quite different: the BCJR has a high sense of (misplaced) confidence, whereas the RNN is more modest in its assessment of its confidence. In the later stages of the decoding, the exaggerated sense of confidence of BCJR leads to an error propagation cascade eventually toggling other bits as well. In this paper we have demonstrated that appropriately designed and trained RNN architectures can'learn' the landmark algorithms of Viterbi and BCJR decoding based on the strong generalization capabilities we demonstrate. This is similar in spirit to recent works on'program learning' in the literature BID14 BID2. In those works, the learning is assisted significantly by a low level program trace on an input; here we learn the Viterbi and BCJR algorithms only by end-to-end training samples; we conjecture that this could be related to the strong "algebraic" nature of the Viterbi and BCJR algorithms. The representation capabilities and learnability of the RNN architectures in decoding existing codes suggest a possibility that new codes could be leant on the AWGN channel itself and improve the state of the art (constituted by turbo, LDPC and polar codes). Also interesting is a new look at classical multi-terminal communication problems, including the relay and interference channels. Both are active areas of present research. The rate-1/2 RSC code introduced in Section 2 is one example of many convolutional codes. In this section, we show empirically that neural decoders can be trained to decode other types of convolutional codes as well as MAP decoder. We consider the following two convolutional codes. Unlike the rate-1/2 RSC code in Section 2, the convolutional code in Figure 8(a) is not recursive, i.e., state does not have a feedback. Also, it is non-systematic, i.e., the message bits can not be seen immediately from the coded bits. The convolutional code in Figure 8 (b) is another type of rate-1/2 RSC code with a larger state dimension (dimension 3 instead of 2). Figure 8 show the architecture of neural network we used for the convolutional codes in Figure 8. For the code in Figure 8(a), we used the exact same architecture we used for the rate-1/2 RSC code in Section 2. For the code in Figure 8 (b), we used a larger network (LSTM instead of GRU and 800 hidden units instead of 400). This is due to the increased state dimension in the encoder. Output For training of neural decoder in Figure 8 (a), we used 12000 training examples of block length 100 with fixed SNR 0dB. For training convolutional code (b), we used 48000 training examples of block length 500. We set batch size 200 and clip norm. The convolutional code (b) has a larger state space. DISPLAYFORM0 Performance. In FIG0 show the BER and BLER of the trained neural decoder for convolutional code in Figure 8 (a) under various SNRs and block lengths. As we can see from these figures, neural decoder trained on one SNR (0dB) and short block length can be generalized to decoding as good as MAP decoder under various SNRs and block lengths. Similarly in FIG0, we show the BER and BLER performances of trained neural decoder for convolutional code in Figure 8 (b), which again shows the generalization capability of the trained neural decoder. Turbo codes, also called parallel concatenated convolutional codes, are popular in practice as they significantly outperform RSC codes. We provide a neural decoder for turbo codes using multiple layers of neural decoder we introduced for RSC codes. An example of rate-1/3 turbo code is shown in FIG0. Two identical rate-1/2 RSC encoders are used, encoder 1 with original sequence b as input and encoder 2 with a randomly permuted version of b as input. Interleaver performs the random permutation. As the first output sequence c 1 of encoder 1 is identical to the output sequence c 1 of encoder 2, and hence redundant. So the sequence c 1 is thrown away, and the rest of the sequences (c 1, c 2, c 2) are transmitted; hence, rate is 1/3.The sequences (c 1, c 2, c 2) are transmitted over AWGN channel, and the noisy received sequences are (y 1, y 2, y 2). Due to the interleaved structure of the encoder, MAP decoding is computationally intractable. Instead, an iterative decoder known as turbo decoder is used in practice, which uses the RSC MAP decoder (BCJR algorithm) as a building block. At first iteration the standard BCJR estimates the posterior Pr(b k |y 1, y 2) with uniform prior on b k for all k ∈ [K]. Next, BCJR estimates Pr(b k |π(y 1), y 2) with the interleaved sequence π(y 1), but now takes the output of the first layer as a prior on b k' s. This process is repeated, refining the belief on what the codewords b k's are, until convergence and an estimation is made in the end for each bit. Training. We propose a neural decoder for turbo codes that we call N-Turbo in FIG0. Following the deep layered architecture of the turbo decoder, we stack layers of a variation of our N-RSC decoder, which we call N-BCJR. However, end-to-end training (using examples of the input sequence SNR for block length 100, 1000, and 10,000 y (i)'s and the message sequence of b (i)'s) of such a deep layers of recurrent architecture is challenging. We propose first training each layer separately, use these trained models as initializations, and train the deep layered neural decoder of N-Turbo starting from these initialized weights. We first explain our N-BCJR architecture, which is a new type of N-RSC that can take flexible bit-wise prior distribution as input. Previous N-RSC we proposed is customized for uniform prior distribution. The architecture is similar to the one for N-RSC. The main difference is input size (3 instead of 2) and the type of RNN (LSTM instead of GRU). To generate N training examples of {(noisycodeword, prior), posterior}, we generate N/12 examples of turbo codes. Then we ran turbo decoder for 12 component decoding -and collect input output pairs from the 12 intermediate steps of Turbo decoder, implemented in python shown in FIG0.We train with codes with blocklength 100 at fixed SNR -1dB. We use mean squared error in as a cost function. To generate training examples with non-zero priors, i.e. example of a triplet (prior probabilities {Pr(b k)} of the intermediate layers, we take as an example the triplet: the input prior probability, the input sequence, and the output of the BJCR layer. We fix training SNR to be -1dB. We stack 6 layers of BCJR decoder with interleavers in between. The last layer of our neural decoder is trained slightly differently to output the estimated message bit and not the posterior probability. Accordingly, we use binary crossentropy loss of as a cost function. We train each N-BCJR layer with 2,000 examples of length 100 turbo encoder, and in the end-to-end training of N-Turbo, we train with 1,000 examples of length 1,000 turbo encoder. We train with 10 epochs and ADAM optimizer with learning rate 0.001. For the end-to-end training, we again use a fixed SNR of noise (-1dB), and test on various SNRs. The choice of training SNR is discussed in detail in the Appendix D.Performance. As can be seen in FIG0, the proposed N-Turbo meets the performance of turbo decoder for block length 100, and in some cases, for test SNR= 2, it achieves a higher accuracy. Similar to N-RSC, N-Turbo generalizes to unseen codewords, as we only show 3, 000 examples in total. It also seamlessly generalizes in the test SNR, as training SNR is fixed at −1dB. In this section, we show the performances of neural networks of various recurrent network architectures in decoding rate-1/2 RSC code and in learning BCJR algorithm with non-zero priors. Table 1 shows the BER of various types of recurrent neural networks trained under the same condition as in N-RSC (120000 example, code length 100). We can see that BERs of the 1-layered RNN and single-directional RNN are order-wise worse than the one of 2-layered GRU (N-RSC), and two layers is sufficient. Table 2 shows the performance of neural networks of various recurrent network As it is natural to sample the training data and test data from the same distribution, one might use the same noise level for testing and training. However, this matched SNR is not reliable as shown in FIG2. We give an analysis that predicts the appropriate choice of training SNR that might be different from testing SNR, and justify our choice via comparisons over various pairs of training and testing SNRs. We conjecture that the optimal training SNR that gives best BER for a target testing SNR depends on the coding rate. A coding rate is defined as the ratio between the length of the message bit sequence K and the length of the transmitted codeword sequence c. The example we use in this paper is a rate r = 1/2 code with length of c equal to 2K. For a rate r code, we propose using training SNR according to DISPLAYFORM0 and call the knee of this curve f (r) = 10 log 10 (2 2r − 1) a threshold. In particular, this gives SNR train = min{SNR test, 0} for rate 1/2 codes. In FIG0 left, we train our neural decoder for RSC encoders of varying rates of r ∈ {1/2, 1/3, 1/4, 1/5, 1/6, 1/7} whose corresponding f (r) = {0, −2.31, −3.82, −4.95, −5.85, −6.59}. f (r) is plotted as a function of the rate r in FIG0 right panel. Compared to the grey shaded region of empirically observed region of training SNR that achieves the best performance, we see that it follows the theoretical prediction up to a small shift. The figure on the left shows empirically observed best SNR for training at each testing SNR for various rate r codes. We can observe that it follows the trend of the theoretical prediction of a curve Table 2: MSE of trained neural models with different number/type of RNN layers in learning BCJR algorithm with non-zero priors with a knee. Before the threshold, it closely aligns with the 45-degree line SNR train = SNR test. around the threshold, the curves become constant functions. We derive the formula in in two parts. When the test SNR is below the threshold, then we are targeting for bit error rate (and similarly the block error rate) of around 10 −1 ∼ 10 −2. This implies that significant portion of the testing examples lie near the decision boundary of this problem. Hence, it makes sense to show matching training examples, as significant portion of the training examples will also be at the boundary, which is what we want in order to maximize the use of the samples. On the other hand, when we are above the threshold, our target bit-error-rate can be significantly smaller, say 10 −6. In this case, most of the testing examples are easy, and only a very small proportion of the testing examples lie at the decision boundary. Hence, if we match training SNR, most of the examples will be wasted. Hence, we need to show those examples at the decision boundary, and we propose that the training examples from SNR 10 log 10 (2 2r − 1) should lie near the boundary. This is a crude estimate, but effective, and can be computed using the capacity achieving random codes for AWGN channels and the distances between the codes words at capacity. Capacity is a fundamental limit on what rate can be used at a given test SNR to achieve small error. In other words, for a given test SNR over AWGN channel, Gaussian capacity gives how closely we can pack the codewords (the classes in our classification problem) so that they are as densely packed as possible. This gives us a sense of how decision boundaries (as measured by the test SNR) depend on the rate. It is given by the Gaussian capacity rate = 1/2 log(1 + SN R). Translating this into our setting, we set the desired threshold that we seek.

Represent the TLDR for finding the corresponding scientific article.

We show that creatively designed and trained RNN architectures can decode well known sequential codes and achieve close to optimal performances.

represent the short summary for finding the corresponding scientific article (long)

We present eligibility propagation an alternative to BPTT that is compatible with experimental data on synaptic plasticity and competes with BPTT on machine learning benchmarks.