Question:
represent the article for finding a single-sentence tldr

Examples:

Given We show that if the usual training loss is augmented by a Lipschitz regularization term, then the networks generalize. We prove generalization by first establishing a stronger convergence , along with a rate of convergence. A second resolves a question posed in: how can a model distinguish between the case of clean labels, and randomized labels? Our answer is that Lipschitz regularization using the Lipschitz constant of the clean data makes this distinction. In this case, the model learns a different function which we hypothesize correctly fails to learn the dirty labels. While deep neural networks networks (DNNs) give more accurate predictions than other machine learning methods BID30, they lack some of the performance guarantees of these other methods. One step towards performance guarantees for DNNs is a proof of generalization with a rate. In this paper, we present such a , for Lipschitz regularized DNNs. In fact, we prove a stronger convergence from which generalization follows. We also consider the following problem, inspired by . Problem 1.1. [Learning from dirty data] Suppose we are given a labelled data set, which has Lipschitz constant Lip(D) = O (see below). Consider making copies of 10 percent of the data, adding a vector of norm to the perturbed data points, and changing the label of the perturbed points. Call the new, dirty, data setD. The dirty data has Lip(D) = O(1/). However, if we compute the histogram of the pairwise Lipschitz constants, the distribution of the values on the right hand side of, are mostly below Lip(D) with a small fraction of the values being O(1/), since the duplicated images are apart but with different labels. Thus we can solve with L 0 estimate using the prevalent smaller values, which is an accurate estimate of the clean data Lipschitz constant. The solution of using such a value is illustrated on the right of Figure 1. Compare to the Tychonoff regularized solution on the right of Figure 2. We hypothesis that on dirty data the solution of replaces the thin tall spikes with short fat spikes leading to better approximation of the original clean data. In Figure 1 we illustrate the solution of (with L 0 = 0), using synthetic one dimensional data. In this case, the labels {−1, 0, 1} are embedded naturally into Y = R, and λ = 0.1. Notice that the solution matches the labels exactly on a subset of the data. In the second part of the figure, we show a solution with dirty labels which introduce a large Lipschitz constant, in this case, the solution reduces the Lipschitz constant, thereby correcting the errors. Learning from dirty labels is studied in §2.4. We show that the model learns a different function than the dirty label function. We conjecture, based on synthetic examples, that it learns a better approximation to the clean labels. We begin by establishing notation. Consider the classification problem to fix ideas, although our restuls apply to other problems as well. Definition 1.2. Let D n = x 1,..., x n be a sequence of i.i.d. random variables sampled from the probability distribution ρ. The data x i are in X = d. Consider the classification problem with D labels, and represent the labels by vertices of the probability simplex, Y ⊂ R D. Write y i = u 0 (x i) for the map from data to labels. Write u(x; w) for the map from the input to data to the last layer of the network.1 Augment the training loss with Lipschitz regularization DISPLAYFORM0 The first term in is the usual average training loss. The second term in the Lipschitz regularization term: the excess Lipschitz constant of the map u, compared to the constant L 0.In order to apply the generalization theorem, we need to take L 0 ≥ Lip(u 0), the Lipschitz constant of the data on the whole data manifold. In practice, Lip(u 0) can be estimated by the Lipschitz constant of the empirical data. The definition of the Lipschitz constants for functions and data, as well as the implementation details are presented in §1.3 below. Figure 1: Synthetic labelled data and Lipschitz regularized solution u. Left: The solution value matches the labels exactly on a large portion of the data set. Right: dirtly labels: 10% of the data is incorrect; the regularized solution corrects the errors. Our analysis will apply to the problem which is convex in u, and does not depend explicitly on the weights, w. Of course, once u is restricted to a fixed neural network architecture, the corresponding minimization problem becomes non-convex in the weights. Our analysis can avoid the dependence on the weights because we make the assumption that there are enough parameters so that u can exactly fit the training data. The assumption is justified by. As we send n → ∞ for convergence, we require that the network also grow, in order to continue to satisfy this assumption. Our apply to other non-parametric methods in this regime. Generalization bounds have been obtained previously via VC dimension analysis of neural networks . The generalization rates have factors of the form A k for a k-layer neural network with bounds w i ≤ A for all weight vectors w i in the network. Such bounds are only applicable for low-complexity networks. Other works have considered connections between generalization and stability BID8 ). More recently, BID5 proposed the Lipschitz constant of the network as a candidate measure for the Rademacher complexity, which is a measure of generalization (, Chapter 26). Also, BID12 showed that Lipschitz regularization can be viewed as a special case of distributional robustness. Unlike other recent contributions such as BID26, our analysis does not depend on the training method. In fact, our analysis has more in common with inverse problems in image processing, such as Total Variation denoising and inpainting BID6 BID36. For further discussion, see Appendix C.The estimate of Lip(u; X) provided by can be quite different from the the Tychonoff gradient regularization BID14, DISPLAYFORM0 since corresponds to a maximum of the values of the norms, and the previous equation corresponds to the mean-squared values. In fact, recent work on semi-supervised learning suggests that higher p-norms of the gradient are needed for generalization when the data manifold is not well approximated by the data BID15 BID10 BID29 BID39. In Figure 2 we compare to the problems in Figure 1 using Tychonoff regularization. The Tychonoff regularization is less effective at correcting errors. The effect is more pronounced in higher dimensions. Figure 2: Synthetic labelled data and Tychonoff regularized solution u. Left: The solution value matches the labels exactly on a large portion of the data set. Right dirty labels: 10% of the data is incorrect; the regularized solution is not as effective at correcting errors. The effect is more pronounced in higher dimensions. An upper bound for the Lipschitz constant of the model is given by the norm of the product of the weight matrices (, Section 4.3). Let w = (w 1, . . ., w J) be the weight matrices for each layer. Then DISPLAYFORM0 Regularization of the network using methods based on has been implemented recently in BID23 and . Because the upper bound in does not take into account the coefficients in weight matrices which are zero due to the activation functions, the gap in the inequality can be off by factors of many orders of magnitude for deep networks BID19.Implementing can be accomplished using backpropagation in the x variable on each label, which can become costly for D large. Special architectures could also be used to implement Lipschitz regularization, for example, on a restricted architecture, BID31 renormalized the weight matrices of each layer to be norm 1.Lipschitz regularization may help with adversarial examples BID40 BID22 which poses a problem for model reliability BID21. Since the Lipschitz constant L ℓ of the loss, ℓ, controls the norm of a perturbation DISPLAYFORM1 maps with smaller Lipschitz constants may be more robust to adversarial examples. BID19 implemented Lipschitz regularization of the loss, and achieved better robustness against adversarial examples, compared to adversarial training BID22 alone. Lipschitz regularization may also improve stability of GANs. 1-Lipschitz networks are also important for Wasserstein-GANs ) BID0. In the gradient penalty away from norm 1 is implemented, augmented by a penalty around perturbed points, with the goal of improved stability. Spectral regularization for GANs was implemented in BID33 ). Definition 1.3 (Lipschitz constants of functions and data). Choose norms · Y, and · X on X and Y, respectively. The Lipschitz constant (in these norms) of a function u: DISPLAYFORM0 When X 0 is all of X, we write Lip(u; X) = Lip(u). The Lipschitz constant of the data is given by DISPLAYFORM1 Finlay & Oberman FORMULA0 implement Lipschitz regularization as follows. The basis for the implementation of the Lipschitz constant is Rademacher's Theorem BID18, §3.1), which states that if a function g(x) is Lipschitz continuous then it is differentiable almost everywhere and DISPLAYFORM2 Restricting to a mini-batch, we obtain the following method for estimating the Lipschitz constant. Let u(x; w) be a Lipschitz continuous function. Then max DISPLAYFORM3 For vector valued functions, the appropriate matrix norm must be used, see §B. The variational problem admits Lipschitz continuous minimizers, but in general the minimizers are not unique. When L 0 = Lip(u 0), it is clear that u 0, is a solution of FORMULA0: both the loss term and the regularization term are zero when applied to u 0. In addition, any L 0 -Lipschitz extension of u 0 | Dn is also a minimizer of, so solutions are not unique. Let u n be any solution of the Lipschitz regularized variational problem. We study the limit of u n as n → ∞. Since the empirical probability measures ρ n converge to the data distribution ρ, the continuum variational problem corresponding to is min DISPLAYFORM0 where in FORMULA8 we have introduced the following notation. Definition 2.1. Given the loss function, ℓ, a map u: X → Y, and a probability measure, µ, supported on X, define DISPLAYFORM1 to be the expectation of the loss with respect to the measure. In particular, the generalization loss of the map u: DISPLAYFORM2 for the average loss on the data set D n, where ρ n:= 1 n δ xi is the empirical measure corresponding to D n. Remark 2.2. Generalization is defined in (, Section 5.2) as the expected value of the loss function on a new input sampled from the data distribution. As defined, the full generalization error includes the training data, but it is of measure zero, so removing it does not change the value. We introduce the following assumption on the loss function. Assumption 2.3 (Loss function). The function ℓ: Y × Y → R is a loss function if it satisfies (i) ℓ ≥ 0, (ii) ℓ(y 1, y 2) = 0 if and only if y 1 = y 2, and (iii) ℓ is strictly convex in y 1.Example 2.4 (R D with L 2 loss). Set Y = R D, and let each label be a basis vector. Set ℓ(y 1, y 2) = y 1 − y 2 2 2 to be the L 2 loss. Example 2.5 (Classification). In classification, the output of the network is a probability vector on the labels. Thus Y = ∆ D, the D-dimensional probability simplex, and each label is mapped to a basis vector. The cross-entropy loss ℓ DISPLAYFORM0 Example 2.6 (Regularized cross-entropy). In the classification setting, it is often the case that the softmax function DISPLAYFORM1 is combined with the cross-entropy loss. In this paper, we regard softmax as the last layer of the DNN, so we assume the output u(x) of the network lies in the probability simplex. If the output, z, of the second to last layer of the DNN, which is the input to softmax in, lies in a compact set, i.e., |z j | ≤ C for all i and some C > 0, then softmax(z) j ≥ e −2C, and so the range of softmax lies in the set DISPLAYFORM2 which is strictly interior to the probability simplex. Restricted to A, the cross-entropy loss ℓ KL is strongly convex and Lipschitz continuous, which is required in Theorems 2.12 and 2.11 below. In our analysis, it is slightly more convenient to define the regularized cross entropy loss with parameter > 0 DISPLAYFORM3 For classification problems, where z = e k, we have ℓ KL (y, e k) = −(1 +) log((y k +)/(1 +)), which is Lipschitz and strongly convex for any 0 ≤ y i ≤ 1 within the probability simplex. Thus, the regularized cross entropy ℓ KL satisfies the strong convexity and Lipschitz regularity required by Theorems 2.12 and 2.11 on the whole probability simplex. Here, we show that solutions of the random variational problem converge to solutions of. We make the standard manifold assumption BID11, and assume the data distribution ρ is a probability density supported on a compact, smooth, DISPLAYFORM0 We denote the probability density again by ρ: M → [0, ∞). Hence, the data D n is a sequence x 1,..., x n of i.i.d. random variables on M with probability density ρ. Associated with the random sample we have the closet point projection map σ n: X → {x 1, . . ., x n} ⊂ X that satisfies DISPLAYFORM1 for all x ∈ X. We recall that W 1,∞ (X; Y) is the space of Lipschitz mappings from X to Y. Throughout this section, C, c > 0 denote positive constants depending only on M, and we assume C ≥ 1 and 0 < c < 1. We follow the analysis tradition of allowing the particular values of C and c to change from line to line. We establish that that minimizers of are unique on M in Theorem A.1, which follows from the strict convexity of the loss restricted to the data manifold M. See also FIG0 which shows how the solutions need not be unique off the data manifold. Our first is in the case where Lip[u 0] ≤ L 0, and so the Lipschitz regularizer is not fully active. This corresponds to the case of clean labels. We state our in generality, for approximate minimizers of, and specialize to the case DISPLAYFORM2 Theorem 2.7 (Convergence ). Assume inf x∈M ρ(x) > 0. For any t > 0, with probability at least 1 − Ct DISPLAYFORM3 is any sequence of minimizers of FORMULA0 and DISPLAYFORM4 and Theorem 2.7 applies to the sequence u n, yielding DISPLAYFORM5 It is important to note that Theorem 2.7 does not requires u n to be minimizers of-we just require zero empirical loss, which is often achieved in practice . This allows for approximation errors in solving on the whole domain X, due to the restriction that u must be expressed via a Deep Neural Network. As an immediate corollary, we can prove that the generalization loss converges to zero, and so we obtain generalization. Corollary 2.9. Assume that for some q ≥ 1 the loss ℓ satisfies DISPLAYFORM6 Then under the assumptions of Theorem 2.7 DISPLAYFORM7 holds with probability at least 1 − Ct −1 n −(ct−1).Proof. By FORMULA21, we can bound the generalization loss as follows DISPLAYFORM8 The proof is completed by invoking Theorem 2.7.We now turn to the proof of Theorem 2.7, which requires a bound on the distance between the closest point projection σ n and the identity. The is standard in probability, and we include it for completeness in Lemma 2.10 proved in §A.1. We refer the interested reader to BID34 for more details. Lemma 2.10. Suppose that inf M ρ > 0. Then for any t > 0 DISPLAYFORM9 with probability at least 1 − Ct −1 n −(ct−1).We now give the proof of Theorem 2.7.Proof of Theorem 2.7. Since L[u n, ρ n] = 0 we have u 0 (x i) = u n (x i) for all 1 ≤ i ≤ n. Thus for any x ∈ X we have DISPLAYFORM10 Therefore, we deduce DISPLAYFORM11 The proof is completed by invoking Lemma 2.10. We now consider the setting of Problem 1.1, illustrated in Figure 1 right. We assume that we only have access to a "dirty" label function, which corresponds to an additive error of the form DISPLAYFORM0 where u clean is the label function, and u e: X → Y is some error function, which is assumed to be zero with high probability. Assume that the error vector e has a much larger Lipschitz constant than the labels, so that Lip(u 0) ≫ Lip(u clean).We wish to fit the clean labels, while not fitting the errors, having access only to u 0. The labels correspond to the subset of the data which generate the low Lipschitz constant L clean, while the errors correspond to pairs of labels that generate a high Lipschitz constant. Thus L clean can easily be estimated from the distribution of the pairwise Lipschitz constants of the data. With the goal in mind, we set L 0 = L clean in. The Lipschitz regularizer is active in, which can lead to the solution succeeding in avoiding the dirty labels, as in Figure 1 right. Our main (Theorems 2.12 and 2.11) show that minimizers of J n converge to minimizers of J almost surely as the number of training points n tends to ∞. It is beyond the scope of this work to estimate to what extent the errors are corrected, however we do know that the solution cannot fit u 0 due to the value of the Lipschitz constant, which is already an improvement over the case λ = 0.The proofs for this section can be found in Section A.2. Theorem 2.11. Suppose that ℓ: Y × Y → R is Lipschitz and strongly convex and let L = Lip(u 0). Then for any t > 0, with probability at least 1 − 2t − m m+2 n −(ct−1) all minimizing sequences u n of and all minimizers u * of satisfy DISPLAYFORM1 The next drops the assumption of strong convexity of the loss. Theorem 2.12. Suppose that inf M ρ > 0, ℓ: Y × Y → R is Lipschitz, and let u * ∈ W 1,∞ (X; Y) be any minimizer of. Then with probability one DISPLAYFORM2 where u n is any sequence of minimizers of. Furthermore, every uniformly convergent subsequence of u n converges on X to a minimizer of. Remark 2.13. In Theorem 2.12 and Theorem 2.11, the sequence u n does not, in general, converge on the whole domain X. The important point is that the sequence converges on the data manifold M, and solves the variational problem off of the manifold, which ensures that the output of the DNN is stable with respect to the input. See FIG0. In this section we provide the proof of stated in §2.3. Theorem A.1. Suppose the loss function satisfies Assumption 2.3. If u, v ∈ W 1,∞ (X; Y) are two minimizers of and DISPLAYFORM0 Therefore, w is a minimizer of J and so we have equality above, which yields DISPLAYFORM1 Since ℓ is strictly convex in its first argument, it follows that u = v on M.Proof of Lemma 2.10 of §2.3. There exists M such that for any 0 < ≤ M, we can cover M with N geodesic balls B 1, B 2,..., B N of radius, where N ≤ C −m and C depends only on M BID25. Let Z i denote the number of random variables x 1,..., x n falling in B i. Then DISPLAYFORM2 m. Let A n denote the event that at least one B i is empty (i.e., Z i = 0 for some i). Then by the union bound we deduce DISPLAYFORM3 In the event that A n does not occur, then each B i has at least one point, and so |x DISPLAYFORM4 The proof of Theorem 2.12 requires a preliminary Lemma. Let DISPLAYFORM5 holds with probability at least 1 − 2t DISPLAYFORM6 The estimate FORMULA0 is called a discrepancy BID41 BID25, and is a uniform version of concentration inequalities. A key tool in the proof of Lemma A.5 is Bernstein's inequality BID7 ), which we recall now for the reader's convenience. For X 1,..., X n i.i.d. with variance DISPLAYFORM7 surely for all i then Bernstein's inequality states that for any > 0 DISPLAYFORM8 Proof of Lemma A.5. We note that it is sufficient to prove the for w ∈ H L (X; Y) with M wρ dV ol(x) = 0. In this case, we have w(x) = 0 for some x ∈ M, and so w L ∞ (X;Y) ≤ CL.We first give the proof for M = X = m. We partition X into hypercubes B 1,..., B N of side length h > 0, where N = h −m. Let Z j denote the number of x 1,..., x n falling in B j. Then Z j is a Binomial random variable with parameters n and p j = Bj ρ dx ≥ ch m. By the Bernstein inequality we have for each j that DISPLAYFORM9 provided 0 < ≤ h m. Therefore, we deduce DISPLAYFORM10 holds with probability at least 1−2h DISPLAYFORM11 holds with probability at least 1 − 2t DISPLAYFORM12 trivially holds, and hence we can allow t > n/ log(n) as well. We sketch here how to prove the on the manifold M. We cover M with k geodesic balls of radius > 0, denoted B M (x 1,),..., B M (x k,), and let ϕ 1,..., ϕ k be a partition of unity subordinate to this open covering of M. For > 0 sufficiently small, the Riemannian exponential map exp x: B(0,) ⊂ T x M → M is a diffeomorphism between the ball B(0, r) ⊂ T x M and the geodesic ball B M (x,) ⊂ M, where T x M ∼ = R m. Furthermore, the Jacobian of exp x at v ∈ B(0, r) ⊂ T x M, denoted by J x (v), satisfies (by the Rauch Comparison Theorem) DISPLAYFORM13 Therefore, we can run the argument above on the ball B(0, r) ⊂ R m in the tangent space, lift the to the geodesic ball B M (x i,) via the Riemannian exponential map exp x, and apply the bound DISPLAYFORM14 to complete the proof. Remark A.6. The exponent 1/(m + 2) is not optimal, but affords a very simple proof. It is possible to prove a similar with the optimal exponent 1/m in dimension m ≥ 3, but the proof is significantly more involved. We refer the reader to BID41 for details. Remark A.7. The proof of Theorem 2.12 shows that Γ-converges to almost surely as n → ∞ in the L ∞ (X; Y) topology. Γ-convergence is a notion of convergence for functionals that ensures minimizers along a sequence of functionals converge to a minimizer of the Γ-limit. While we do not use the language of Γ-convergence here, the ideas are present in the proof of Theorem 2.12. We refer to BID9 for details on Γ-convergence. Proof of Theorem 2.12. By Lemma A.5 the event that lim DISPLAYFORM15 for all Lipschitz constants L > 0 has probability one. For the rest of the proof we restrict ourselves to this event. Let u n ∈ W 1,∞ (X; Y) be a sequence of minimizers of, and let u * ∈ W 1,∞ (X; Y) be any minimizer of. Then since DISPLAYFORM16 we have Lip(u n) ≤ Lip(u 0) =: L for all n. By the Arzelà-Ascoli Theorem BID37 there exists a subsequence u nj and a function u ∈ W 1,∞ (X; Y) such that u nj → u uniformly as n j → ∞. Note we also have Lip(u) ≤ lim inf j→∞ Lip(u nj). Since DISPLAYFORM17 Therefore, u is a minimizer of J. By Theorem A.1, u = u * on M, and so u nj → u * uniformly on M as j → ∞. Now, suppose that does not hold. Then there exists a subsequence u nj and δ > 0 such that DISPLAYFORM18 for all j ≥ 1. However, we can apply the argument above to extract a further subsequence of u nj that converges uniformly on M to u *, which is a contradiction. This completes the proof. Proof of Theorem 2.11. Let L = Lip(u 0). By Lemma A.5 DISPLAYFORM19 (FORMULA0 holds with probability at least 1 − 2t − m m+2 n −(ct−1) for any t > 0. Let us assume for the rest of the proof that holds. (LE) is not practical for large scale problems. There has be extensive work on the Lipschitz Extension problem, see, BID28, for example. More recently, optimal Lipschitz extensions have been studied, with connections to Partial Differential Equations, see BID2. We can interpret as a relaxed version of (LE), where λ −1 is a parameter which replaces the unknown Lagrange multiplier for the constraint. Variational problems are fundamental tools in mathematical approaches to image processing BID3 Lipschitz regularization in not nearly as common. It appears in image processing in (, §4.4) BID16 and BID24 ). Variational problems of the form can be studied by the direct method in the calculus of variations BID13. The problem can be discretized to obtain a finite dimensional convex convex optimization problem. The variational problem can also be studied by finding the first variation, which is a Partial Differential Equation BID17, which can then be solved numerically. Both approaches are discussed in BID3. In FIG1 we compare different regularization terms, in one dimension. The difference between the regularizers is more extreme in higher dimensions. it matches with We prove generalization of DNNs by adding a Lipschitz regularization term to the training loss. We resolve a question posed in Zhang et al. (2016). but not with This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.

Question: Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since the synchronization of the local models or gradients can be a bottleneck for large-scale distributed training, compressing communication traffic has gained widespread attention recently. Among several recent proposed compression algorithms, Residual Gradient Compression (RGC) is one of the most successful approaches---it can significantly compress the transmitting message size (0.1% of the gradient size) of each node and still preserve accuracy. However, the literature on compressing deep networks focuses almost exclusively on achieving good compression rate, while the efficiency of RGC in real implementation has been less investigated. In this paper, we develop an RGC method that achieves significant training time improvement in real-world multi-GPU systems. Our proposed RGC system design called RedSync, introduces a set of optimizations to reduce communication bandwidth while introducing limited overhead. We examine the performance of RedSync on two different multiple GPU platforms, including a supercomputer and a multi-card server. Our test cases include image classification on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets. For DNNs featured with high communication to computation ratio, which has long been considered with poor scalability, RedSync shows significant performance improvement. For training large-scale deep neural networks (DNNs) on multiple computing nodes, data parallelism has emerged as the most popular choice due to its simplicity and effectiveness BID5; BID13 ). However, the communication bandwidth of network fabric has become the bottleneck limiting data parallel performance. On one hand, models of DNNs, which already contain tens to hundreds of layers and totaling 10-20 million parameters today, continue to grow bigger. Therefore, the requirement of communicating model parameter updates among all computing nodes poses a higher challenge to network bandwidth. On the other hand, the development of DNN training accelerators has shifted the bottleneck of training towards communication across models. As the evolution of the inter-connected network bandwidth is not as fast as computing hardware, synchronization overhead has become the bottleneck of data parallelism on distributed systems using new computing hardware. Many recent studies focused on reducing the communication cost between nodes by reducing the size of the gradients to be transmitted. One line of work BID15; BID2; ) propose to quantize the gradients to low-precision values. Considering compression ratio (ratio of compressed gradients size to their original size) achieved by quantization is limited, another line of research orthogonal to quantization is to sparsify communication gradients and restrict weight-updates to a small subset of parameters. Residual Gradient Compression (RGC) method (; BID0 ; BID4 ; BID9 ; BID14) is currently the most promising pruning method to achieve good compression ratio while ensuring no loss of training accuracy. It transmits only a small subset of gradients and maintains the remaining gradients locally as residuals to be added to gradients of the next iteration. The first RGC implementation is proposed by and uses a threshold-based method to only send gradients larger than a predefined constant threshold for fully-connected layers. Considering a predefined threshold is hard to be chosen appropriately, BID0 improve the robustness of RGC by selecting top 1% gradients to communicate according to their magnitude. Because these two implementations are tuned for some specific network structures, applying them to other DNNs will lead to accuracy loss as indicated in BID4. Based on their work, the latest RGC variants, such as BID14; BID4; BID9 ), are able to achieve a 0.1% compression ratio on local gradients while ensuring almost no loss of model accuracy on a variety of DNN structures after introducing some key modifications. Despite of good model accuracy achieved with simulation experiments, no recent studies have discussed the potential performance gain after integrating the latest RCG methods to real distributed training system, especially to the multi-GPU systems equipped with high-quality network infrastructures. The challenges of applying RGC to distributed GPU systems come from two aspects. First, there is no efficient compression algorithm proposed for RGC method. According to our experimental , selecting top-0.1% elements with the state-of-the-art GPU-based top-k algorithm are so expensive that the overhead of compression is much higher than the benefits of network bandwidth reduction. Second, synchronization of sparse data structures is nontrivial to be supported with existing efficient communication libraries, such as Message Passing Interface (MPI), which are designed for dense data structures. Targeting multi-GPU systems, a highly-efficient RGC implementation called RedSync is proposed. Our contributions are listed as follows:• We combined pruning and quantization techniques together to compress transmitting gradients. A set of parallel-friendly top-0.1% selection methods are designed to support pruning operations inside GPU device memory, which are orders of magnitude faster than the stateof-the-art GPU-based top-k selection method.• Considering the distribution characteristics of communication data, we apply allgather operation using MPI for a sparse synchronization scheme. A cost model is derived to analyze both communication cost and calculation overhead. Based on it, we pointed out potential performance gain and the bottleneck of our implementation.• RedSync is able to ensure almost no accuracy loss to train a set of DNNs after integrating with the latest algorithm improvements. This is the first work, as far as we known, to evaluate the performance of RGC method on the scale of 128 GPUs. RedSync provides significant performance improvements for communication-intensive networks, like VGG, AlexNet and some LSTMs. Input: node id k, the number of node N Input: dataset χ Input: mini batch size b per node Input: DISPLAYFORM0; w) by forward and backward propagation for j = #layer, #layer − 1,..., 0 do V DISPLAYFORM1 We first give an overview of a simple RGC workflow used in RedSync (see more details in Algorithm 1). We denote a DNN model as f (w), where w is the vector of parameters. We assume a system has N workers. Each worker, say the k-th worker, holds a local dataset χ t k at iteration t with size b and a local copy of the global weight w. Synchronous SGD method is adopted in RedSync. At each iteration, node k computes the gradient G k using local data, where G k j indicates gradients of layer j. Each node also maintains a residual V k, which is initialized as 0 and used to accumulate untransmitted gradient from previous iterations. After added with latest gradient, a subset of residuals is selected as the communication-set, and is compressed into sparse data structures. The select operation in Algorithm 1 chooses more important elements based on magnitude. Those selected elements (denoted ask Masks) are synchronized among all the nodes using allreduce operations, which is able to take advantage of the highly-optimized allreduce operation on HPC systems (Figure 1: Performance of four communication-set selection methods under message sizes. Elements in the data list are generated randomly from a standard uniform distribution. Comm. illustrates the time taken to synchronize the message through a network with a peak bandwidth of 3.5GBps by allreduce operation. Performance is measured as total time cost for 100 times independent operations. implemented with allreduce has been widely adopted in state-of-the-art large-scale CNN training tasks BID7 and). Remaining elements outside the communicationset are assigned as new residuals of the next iteration. The workflow of this algorithm is the same as an RGC variant called Deep Gradient Compression Method mentioned BID9. In the following, we details our contribution in implementations of select, Allreduce and decompress to make this workflow efficient in practice. The efficiency of communication-set selection method is critical for the RGC system's overall performance. Since a predefined threshold is difficult to determine, recent work BID9; BID14 ) suggest to select top 0.1% elements from residuals of each layer as the communication-set. However, the top-0.1% selection is nontrivial to be implemented on GPU. One of the most efficient top-k selection methods designed for GPU can be implemented based on radixSelect algorithm BID1 ), which determines each bit of the k-th largest element by scan and scatter. Serial scan BID17 ) and scatter operations are extremely timeconsuming. As shown in Figure 1, the computation time for top-0.1% with radixSelect on a Titan X GPU sometimes is even slightly higher than the time for synchronizing these parameters through a 3.5 GBps network. To avoid performing a top-0.1% operation on a large number of parameters, we propose two communication-set selection algorithms called trimmed top-k selection and threshold binary search selection, which are more efficient on GPUs. Trimmed top-k selection. Observing that the distribution of residuals is usually similar to a normal distribution, we can use statistical features to remove most of the smaller elements and limit radixSelect operation on a relatively small subset. As shown in Algorithm 2, we first calculate the mean and maximum of residuals' absolute values of this layer. A relative large threshold value is chosen according to mean and maximum value, for example, 0.8 × (max − mean) + mean. Operation count nonzero gets the number of elements whose absolute values are greater than the threshold. If the number is smaller than k (the number of top-0.1% elements), we dynamically decrease the threshold until we find the number of parameters whose absolute value above the threshold is larger than k. Then we trim all elements that are less than the threshold and perform a top-k selection operation using radixSelect on the remaining elements. Operation mean, max and count nonzero can all be efficiently implemented with a single reduction operation. nonzero indices is a typical stream compaction problem, which uses just one scan operation as its backbone BID16 ).Threshold binary search selection. For some layers with very large numbers of parameter elements, even conducting radixSelect on a small subset of elements will still be a very time-consuming operation. In order to completely avoid using radixSelect operation on GPU, we propose a method to select approximate top-0.1% elements as communication-set. Instead of identifying the kth (top 0.1%th) largest element, we search for a threshold to make it between the kth to 2kth largest element, and then select elements larger than the threshold as communication-set. In this case, at least 0.1% largest elements are included in the communication-set. As shown in Algorithm 3, we use a binary search algorithm to find such a threshold. To avoid excessive searching, it will always be terminated when the difference of left bound and right bound is less than a small value. Input: tensor to be compressed X Input: number of elements remained k Output: < indice, values > 1: mean ← mean(abs(X)) 2: max ← max(abs(X)) 3: ← 0.2 4: ratio ← (1 −) 5: nnz = count nonzero(abs(X) > threshold) 6: while nnz > k do 7: threshold ← mean + ratio × (max − mean) 8: nnz = count nonzero(abs(X) > threshold) 9: ratio = ratio − 10: end while 11: DISPLAYFORM0 Algorithm 3 Top-k selection with threshold binary search selection Input: tensor to be compressed X Input: number of elements remained k Input: Termination condition parameter Output: < indice, values > 1: mean ← mean(abs(X)); max ← max(abs(X)) 2: l ← 0.0; r ← 1.0; threshold = 0.0 3: while r − l > do 4: ratio = l + (r − l)/2 5: threshold ← mean + ratio × (max − mean) 6: nnz = count nonzero(abs(X) > threshold) 7: if nnz > k and 2k > nnz then 8: For layers with large sizes, such as the first fully-connected layer in VGG16 and softmax layer in LSTM, the time for count nonzero operation is still not negligible. We further improve the efficiency of the selection algorithm by reducing the number of count nonzero operations. We recommend that, after a threshold binary search for this layer, the threshold element can be reused in the next few iterations. The interval of search is empirically set to 5, and the selection algorithm introduces only one nonzero count overhead on average. In Figure 1, we compared the time cost of different selection approaches on parameter lists of different sizes. Compared with directly performing radixSelect, both proposed methods significantly reduce the selection time for large sizes. For top-0.1% selection on 64MB elements, trimmed top-k and sampled threshold binary search selection are 38.13 and 16.17 × faster than radixSelect. In practice, we dynamically choose compression strategies: For smaller parameter sets such as biases and batch norm layers, we do not compress residuals or directly use radixSelect to select top-0.1% significant elements. Trimmed top-k selection is suitable for parameters of middle size layers, like convolutional layers, because it can ensure the compression ratio to be exactly 0.1% and introduce no extra communication bandwidth requirements. Threshold binary search based selection is suitable for large size layers, like hidden layers and softmax layers in LSTMs, for which the compression cost is more critical to be optimized than the communication cost. Compressed residuals should include k indices and k values. We further investigate the possibility of quantizing these values. By setting the values of all elements of the same sign in the communicationset to their mean, we can almost eliminate the communication bandwidth requirement of value information transmitting by using only one floating-point number instead of k. In order to facilitate quantization compression, we slightly modify our select method to ensure that elements in the communication-set are all of the same sign. It can be achieved by choosing the largest k elements and the smallest k elements as communication-set in turns. In other words, if we select the largest k elements (all positive numbers) in this layer as the communication-set at current iteration, we will choose smallest k elements (all negative numbers) as the communication-set for the next iteration. It is worth noting that sampled threshold binary search selection cannot be used with quantization. In addition, we do not quantify the output layer of the DNN, in order to distinguish the correct classification information. Synchronization of dense gradient structures in traditional distributed DNN systems can be simply implemented with an allreduce operation, which has been well-studied on multiple-GPU systems BID3 ). However, the design of a sparse allreduce in a distributed setting is not as simple because each worker may contribute different non-zero indices in its compressed residuals. According to our observation, there are very few overlapping indices of the communication-set distribution of different nodes. For example, training VGG16 on Cifar10 dataset using 16 GPUs with a compression ratio as 0.1% for each node, the averaged compression ratio of synchronized residuals of all nodes is 1.55%. We utilize the allgather operation, an operation in which the data contributed by each node is gathered at all nodes, to implement sparse allreduce. The message representing compressed residuals of each node should include the information of indices and values of elements in communication-set. When using threshold binary search selection, the length of each node's message is different. As a , the packaged message should also include an initial element, which indicates the length of the compressed elements. Instead of using two allgather operations for indices and values message separately, we package the indices and values into a single message to reduce latency. After finishing the allgather operation, each node collects N compressed residuals of this layer from all the other nodes. We add the compressed residuals to the corresponding weights in the local model after scaling with the learning rate. It can be seen as an operation that adds a sparse array to a dense array, which has been fully-optimized in Level 1 function axpyi of cuSparse library on GPU. RedSync implements a set of algorithm improvement techniques proposed in BID9. We details momentum correction, momentum factor masking and our modification to warmup training in Appendix C, as well as local gradient clipping in Appendix B. To analyze the potential performance gain of sparse synchronization, we adopt a widely-used performance model to estimate the communication cost in terms of latency and bandwidth used. We assume that the time taken to send a message between any two nodes can be modeled as α + nβ, where α is the latency (or startup time) per message, independent of message size, β is the transfer time per byte, and n is the number of bytes transferred. The node's network interface is assumed to be single ported; i.e. at most one message can be sent and one message can be received simultaneously. M is the number of elements in residuals of current layer. D is the compression ratio. In the case of reduction operations, we assume that γ 2 is the computational cost for performing the reduction operation for a message of size M, and γ 1 is the cost to decompress the collected sparse message of size M. For the case where the compression ratio of each node is different, which is always true for the binary search method, D represents the average compression ratio of all nodes. Suppose that we use recursive doubling for allgather and Rabenseifners algorithm mentioned in for allreduce communication. The cost of quantized sparse and dense synchronization is illustrated Equation 1 and 2, respectively. The derivations are left in Appendix A. DISPLAYFORM0 As implicated by the performance model, the compression rate for the model is not equal to the compression rate for communication bandwidth. The bandwidth term of sparse synchronization is (p − 1)DM β, which is proportional to the number of nodes p. Even if the sparseness D is 0.1% for all p node, when p is 128, the communication bandwidth for sparse synchronization will be 12.8% of dense synchronization rather than 0.1% of dense synchronization. Second, the overhead of reduction may be a new bottleneck when scaling RedSync to larger scale. The last term pγ 1 in Eq. 1 indicates that the overhead to do reduction also increases linearly with the number of nodes p. However, in Eq. 2, reduction overhead almost does not increase with number of nodes. We tested the accuracy and performance of our proposed implementation on two different multi-GPU systems, including a world's top GPU supercomputer and a multi-GPU server. Muradin is a server with eight GPUs in the same node. It is equipped with one Intel(R) Xeon(R) CPU E5-2640 v4 and 8 TITAN Vs, which is connected to the CPU through PCI-E 3.0. Piz Daint is a GPU supercomputer. Each node of it includes two Intel Xeon E5-2690v3 CPUs and one NVIDIA Tesla P100 GPUs. In total, there are 5320 nodes connected by Aries interconnect with Dragonfly topology. We used pytorch v4.0 to conduct basic DNN training operations. For communication library, horovod an MPI wrapper upon pytorch, is used to provide collective communication operations. Horovod was compiled with OpenMPI v3.1 with cuda-aware supported on both systems. We tested our performance on two major types of mainstream deep learning applications. For Image Classification tasks, we studied ResNet-44 and VGG16 on Cifar10 BID8 ), AlexNet, VGG16 and ResNet-50 on ImageNet BID6 ). For all CNNs, we used Nesterov's momentum SGD as optimizer. We used the same learning rate strategies as the SGD for the RGC methods. Warm-up technique was applied to the first 5 epochs of ResNet50 and VGG16 for both SGD and RGC. For Language Modeling tasks, we picked two datasets for evaluation. The Penn Treebank corpus (PTB) dataset consists of 923,000 training, 73,000 validation and 82,000 test words BID10 ). The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia BID11 ). It consists 2,088,628 training, 217,646 and 245,569 test words. We adopted a 2-layer LSTM language model architecture with 1500 hidden units per layer BID12 ) to evaluate both datasets. We tied the weights of encoder and decoder and use vanilla SGD with gradient clipping. Learning rate decays when no improvement has been made in validation loss. We examined the convergence of RedSycn on the datasets mentioned before. For the Cifar10 dataset, we used two CNNs, i.e. ResNet44 and VGG16, as test cases. Both DNNs were tested on 4 GPUs, and the total mini-batch size is 256. On the ImageNet dataset, we tested AlexNet, ResNet50, and VGG16. On the PTB and Wiki2 dataset, we examined the perplexity of the 2-layer LSTM mentioned before. FIG1 shows the validation error of RGC and quantized RGC provided by RedSync on three test cases compared with original SGD. More comprehensive are shown in the left side of Table 1. We also tested the sensitivity of the RGC method to large training data batch size. As shown in the right side of Table 1 when increasing the batch size to 2048, RedSync got no loss of accuracy compared to the original SGD. Next we tested the performance and scalability of RedSync as number of GPUs grow. Fig. 5 illustrates scalability of RedSync on Piz Daint with four test cases. Fig. 3 and Fig. 4 show the performance of RedSync on Muradin with six test cases. We compared RedSync and its quantization version Quantized-RedSync with a baseline data parallel implementation provided by horovod. Data was collected by averaging training time in 1000 training iterations. We used trimmed top-k algorithm to compress layers in CNNs larger than 128KB and used threshold binary search algorithm for hidden layers and the softmax layer for LSTM. Fig. 6 illustrates the cost of different parts using RedSync when scaling it to 128 GPUs on Piz Daint. Our observations are summarized as follows.1. Using our parallel-friendly selection methods for compression is critical for system overall performance. In Fig. 3 and Fig. 4, we added an RGC implementation called pure RGC, which uses radixSelect to select top 0.1% elements as communication-set rather than our proposed methods. The performance of pure RGC is even slower than the baseline version, because compression time is too long.2. RedSync is suitable for accelerating data parallel training on DNNs with high communication to computation ratio. For VGG16, AlexNet and LSTM, although performance of RedSync on a single GPU is not as good as baseline version due to compression and decompression overhead, RedSync can achieve significant speedup with more than 2 GPUs. However, we observed no performance gain for ResNet50 both on Piz Daint and Muradin. As implicated in Table 1, the ratio of computation to communication of ResNet50 is the highest in the DNNs we investigated. On large scale, most of time during ResNet50 training with RedSync is wasted on decompression phase, as shown in Fig. 6, which overdrafts the benefit of communication bandwidth reduction.3. The scalability curve of RedSync on Piz Daint shows a concave shape. For example, as shown in Fig. 5, RedSync gets a better speedup to baseline version on 32 GPUs than 128 GPUs for AlexNet. It is because that communication bandwidth requirement and decompression overhead both grow linearly with the number of GPU in use. Such phenomenon verifies our analysis using communication performance model. Quantized-RedSync always achieves better performance than RedSync for CNNs. However, for LSTM training on small scale, Quantized-RedSync achieves worse performance than RedSync. This is due to the balance of communication and computational overhead. CNN adopts trimmed top-k as the communication-set selection method and its quantized version has similar computation cost. As shown in Fig. 6, no significant difference of selection cost in CNN training. Therefore, the reducing of communication cost by quantization improves the system's overall performance. As for LSTMs, they use sampled threshold binary search as selection for non-quantized RedSync, but use threshold binary search for quantized RedSync. Sampled selection is much more faster. Therefore, on small-scale, RedSync has better performance than Quantized-RedSync due to less selection overhead. When scaling to more than 16 GPUs, benefit from the reduction of communication compensates for the cost of the communication-set selection. This paper proposes a distributed implementation called RedSync to accelerate data parallel DNN training by utilizing a type of gradient sparsification method named as Residual Gradient Compression (RGC). We solved two major obstacles to implement RGC on multi-GPU systems: high overhead of compression using GPU and lack of support for collective communication implementation for sparse data structures. We tested the performance of RedSync on two GPU platforms, including a supercomputer system and a multi-GPU server. For AlexNet, VGG16, and LSTM, we observed significant speedup for large-scale DNN training. The left part of FIG3 illustrates how sparse allgather works by recursive doubling method. We assume the compression rate on all of the node is the same as D. If we use threshold binary search for communication-set selection, D here should be the average compression ratio of all nodes for a good approximation. In the first step, nodes that are a distance 1 apart exchange their compressed residuals, the size of which is M × D. In the second step, nodes that are a distance 2 apart exchange their own data as well as the data they received in the previous step, which is 2M × D in total. In the third step, nodes that are a distance 4 apart exchange their own data as well the data they received in the previous two steps. In this way, for a power-of-two number of processes, all processes get all the data in lgp steps. The amount of data exchanged by each node is M × D in the first step, 2M × D in the second step, and so forth, up to 2 lg(p)−1 M × D in the last step. Therefore, The time for message transfer taken by this algorithm is T transf er = lg(p)α + (p − 1)M × Dβ. After including decompressing overhead γ for collected p different compressed residuals and communication selection overhead T select, the time for all-gather based synchronization should be T transf er DISPLAYFORM0 As shown in the right part of FIG3, the Rabenseifners algorithm is adopted for allreduce operation on messages. It does a reduce-scatter followed by an allgather. Reduce-scatter is a variant of reduce in which the , instead of being stored at the root, is scattered among all p nodes. We use a recursive halving algorithm, which is analogous to the recursive doubling algorithm used for allgather but in reverse way. In the first step, each node exchanges data with a node that is a distance p/2 away: Each process sends the data needed by all processes in the other half, which is of size M/2. They also receives the data needed by all processes in its own half, and performs the reduction operation on the received data. In the second step, each process exchanges data with a process that is a distance p/4 away. This procedure continues recursively, halving the data communicated at each step, for a total of lgp steps. After reduce-scatter, allgather phase will have the the same bandwidth and latency requirements. The time taken by Rabenseifners algorithm is the sum of the times taken by reduce-scatter (recursive halving), allgather and reduction operations. The total time should be DISPLAYFORM1 It is necessary to improve data parallel efficiency by overlapping communication with computation through pipelining communication and gradient calculation. Before updating aggregated gradients after scaling with learning rate to weights, gradient clipping is usually adopted to avoid gradient explosion. It rescales all of the gradients when the sum of their norms exceeds a threshold. For RGC methods, the local clipping technique BID9 ) is adopted to perform gradient clipping by a new threshold (N −1/2 of original) locally before adding the current gradients to previous residuals. The difference is that traditional data parallel does clipping after communication of all layers are completed, while the RGC algorithm needs to do clipping before communication. In this case, we need to wait for the completion of the entire back-propagation to get gradients of all layers. And then we do clipping on gradients and then perform compression for communication. Local clipping is equivalent to introducing synchronization between computing and communication and thus eliminating the possibility of Communication hiding. As shown in FIG4, We have abandoned gradient clipping for CNNs, which seldom have gradient exploration problem for the deep networks in order to explore the potential overlapping. As for RNNs, gradients are achieved after backpropagation of all time steps using Back Propagation Through Time (BPTT). When backpropagation of the last layer is completed, we use the gradients of all layers to conduct local gradient clipping. In this case, the communication time can only overlap with the compression calculation. Because even with the original data parallel approach, the computation and communication overlap for each layer can only be made at the last time step, RGC dose not introduce too much overhead. We integrate the momentum masking and momentum correction schemes as proposed in BID9 for momentum SGD and Nesterov momentum SGD optimizers in RedSync. The Momentum SGD version of RGC method adopted by RedSync is illustrated in Algorithm 4. A warm-up training, by exponentially decreasing the compression ratio of the residuals in communication-set in first few epochs, is generally adopted to accelerate convergence in the first few iterations. For example, it is recommended to decrease the compression ratio of residuals in the warm-up period as follows: 25%, 6.25%, 1.5625%, 0.4%, 0.1%. However, we find it could be inefficient for large-scale. As analyzed in the previous section, even synchronization of compressed residual with a compression ratio as 1.5625% requires 100% bandwidth of dense allreduce for quantized RedSync on 64 GPUs. Instead of adopting high-compression-ratio RGC method of warm-up training, we use original SGD optimizer synchronized by allreduce in first few epochs if necessary.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We proposed an implementation to accelerate DNN data parallel training by reducing communication bandwidth requirement.

We extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective, and we show that, given a desired FLOPs requirement, different neural networks are successfully trained.


Question: Represent the scientific article to find a short TLDR

Question: Backpropagation is driving today's artificial neural networks. However, despite extensive research, it remains unclear if the brain implements this algorithm. Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative. However, the convergence rate of such learning scales poorly with the number of involved neurons. Here we propose a hybrid learning approach, in which each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide. We show that our approach learns to approximate the gradient, and can match the performance of gradient-based learning on fully connected and convolutional networks. Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules. It is unknown how the brain solves the credit assignment problem when learning: how does each neuron know its role in a positive (or negative) outcome, and thus know how to change its activity to perform better next time? Biologically plausible solutions to credit assignment include those based on reinforcement learning (RL) algorithms. In these approaches a globally distributed reward signal provides feedback to all neurons in a network. However these methods have not been demonstrated to operate at scale. For instance, variance in the REINFORCE estimator scales with the number of units in the network. This drives the hypothesis that learning in the brain must rely on additional structures beyond a global reward signal. In artificial neural networks, credit assignment is performed with gradient-based methods computed through backpropagation. This is significantly more efficient than RL-based algorithms. However there are well known problems with implementing backpropagation in biologically realistic neural networks. For instance backpropagation requires a feedback structure with the same weights as the feedforward network to communicate gradients (so-called weight transport). Yet such structures are not observed in neural circuits. Despite this, backpropagation is the only method known to solve learning problems at scale. Thus modifications or approximations to backpropagation that are more plausible have been the focus of significant recent attention. Notably, it turns out that weight transport can be avoided by using fixed, random feedback weights, through a phenomenon called feedback alignment. However feedback alignment does not work in larger, more complicated network architectures (such as convolutional networks). Here we propose to use an RL algorithm to train a feedback system to enable learning. We propose to use a REINFORCE-style perturbation approach to train a feedback signal to approximate what would have been provided by backpropagation. We demonstrate that our model learns as well as regular backpropagation in small models, overcomes the limitations of fixed random feedback weights ("feedback alignment") on more complicated feedforward networks, and can be utilized in convolutional networks. Our method illustrates a biologically realistic way the brain could perform gradient descent-like learning. Let an N hidden-layer network be given byŷ = f (x) ∈ R p, composed of a set of layer-wise summation and non-linear activations, for hidden layer states h i ∈ R ni, non-linearity σ and with input h 0 = x and output h N +1 =ŷ. Define L as the loss function L(x, y), where the data (x, y) ∈ D are drawn from a distribution ρ. Our aim is then to minimize: E ρ [L(x, y)]. Backpropagation computes the error signalẽ i in a top-down fashion: Let the loss gradient term be denoted as Here we replace λ i with an approximation, with its own parameters to be learned:, for parameters B. We will useẽ i to denote the gradient signal backpropagated through the synthetic gradients, and e i for the true gradients. To estimate B we use stochasticity inherent to biological neural networks. For each input each unit produces a noisy response: ) with standard deviation c h > 0. This then generates a noisy lossL(x, y, ξ) and a baseline loss L(x, y) =L(x, y, 0). We will use the noisy response to estimate gradients, that then allow us to optimize the baseline L. This is achieved by linearizing the loss: To demonstrate the method can be used to solve simple supervised learning problems we use node perturbation with a four-layer network and MSE loss to solve MNIST (Fig. 1). We approximate loss gradients as follows: Tẽi+1. The feedback parameters B i+1 are estimated by solving the least squares problem:, whereλ is the perturbationbased estimator derived above. B is updated with each mini-batch using stochastic gradient-descent to minimize this loss. 1 Updates to W i are made using the synthetic gradients ∆W i = ηẽ i h i−1, for learning rate η. We observed that the system is able to provide a close correspondence between the feedforward and feedback matrices in both layers of the network (Fig. 1a). The relative error between B i and W i is lower than what is observed for feedback alignment, suggesting that this co-adaptation of W i and B i is indeed beneficial. We observe that the alignment (the angle between the estimated gradient and the true gradient, proportional to e T W B) is lower for node perturbation than for feedback alignment (Fig. 1b). Recent studies have shown that sign congruence of the feedforward and feedback matrices is all that is required to achieve good performance. Here the sign congruence is also higher in node perturbation (Fig. 1c). Finally, the learning performance of node perturbation is comparable to backpropagation (Fig. 1d) -achieving close to 3% test error. These suggest node perturbation for learning feedback weights can be used in deep networks. Hyperparameters found through random search. A known shortcoming of feedback alignment is in auto-encoding networks with tight bottleneck layers. To see if our method has the same shortcoming we examine a simple auto-encoding network with MNIST input data (size 784-200-2-200-784, MSE loss). We also compare the method to the'matching' learning rule, in which updates to B match updates to W. As expected, feedback alignment performs poorly. Node perturbation actually performs better than backpropagation, and comparable to ADAM (Fig. 2a). In fact ADAM begins to overfit, while node perturbation does not. The matched learning rule performs similarly to backpropagation. These are surprising at first glance. Perhaps, similar to feedback alignment, learning feedback weights strikes the right balance between providing a useful signal to learn, and constraining the updates to be sufficiently aligned with B, acting as a type of regularization. The noise added when estimating the feedback weights may also serve to regularize the latent representation, as, indeed, the latent space learnt by node perturbation shows a more evenly distributed separation of digits. While, in contrast, the representations learnt by backprop and ADAM show more structure, and feedback alignment does not learn a useful representation at all (Fig. 2b,c). These show that node perturbation is able to successfully communicate error signals through thin layers of a network as needed. Finally we test the method on a convolutional neural network (CNN) solving CIFAR10. The CNN has the architecture Conv(3x3, 1x1, 32), MaxPool(3x3, 2x2), Conv(5x5, 1x1, 128), MaxPool(3x3, 2x2), Conv(5x5, 1x1, 256), MaxPool(3x3, 2x2), FC 2048, FC 2048, Softmax, with hyperparameters found through random search. For this network we learn feedback weights direct from the output layer to each earlier layer: TẽN (similar to 'direct feedback alignment'). Here this was solved by gradient-descent. We obtain a test accuracy of 75.2%. When compared with fixed feedback weights (test accuracy of 72.5%) and backpropagation (test accuracy of 77.2%), we see it is advantageous to learn feedback weights. This shows the method can be used in a CNN, and can solve challenging computer vision problems without weight transport. Here we implement a perturbation-based synthetic gradient method to train neural networks. We show that this hybrid approach can be used in both fully connected and convolutional networks. By removing both the symmetric feedforward, feedback weight requirement imposed by backpropagation this approach is a step towards more biologically-plausible deep learning. In contrast to many perturbation-based methods, this hybrid approach can solve large-scale problems. We thus believe this approach can provide powerful and biologically plausible learning algorithms. While previous research has provided some insight and theory for how feedback alignment works the effect remains somewhat mysterious, and not applicable in some network architectures. Recent studies have shown that some of these weaknesses can be addressed by instead imposing sign congruent feedforward and feedback matrices. Yet what mechanism may produce congruence in biological networks is unknown. Here we show that the shortcomings of feedback alignment can be addressed in another way: the system can learn to adjust weights as needed to provide a useful error signal. Our work is closely related to Akrout et al 2019, which also uses perturbations to learn feedback weights. However our approach does not divide learning into two phases, and training of the feedback weights does not occur in a layer-wise fashion. Here we tested our method in an idealized setting, however it is consistent with neurobiology in two important ways. First, it involves the separate learning of feedforward and feedback weights. This is possible in cortical networks where complex feedback connections exist between layers, and where pyramidal cells have apical and basal compartments that allow for separate integration of feedback and feedforward signals. Second, noisy perturbations are common in neural learning models. There are many mechanisms by which noise can be measured or approximated, or neurons could use a learning rule that does not require knowing the noise. While our model involves the subtraction of a baseline loss to reduce the variance of the estimator, this does not affect the expected value of the estimator; technically the baseline could be removed or approximated. Thus we believe our approach could be implemented in neural circuits. There is a large space of plausible learning rules that can learn feedback signals in order to more efficiently learn. These promise to inform both models of learning in the brain and learning algorithms in artificial networks. Here we take an early step in this direction. We review the key components of the model. Data (x, y) ∈ D are drawn from a distribution ρ. The loss function is linearized: such that with expectation taken over the noise distribution ν(ξ). This suggests a good estimator of the loss gradient iŝ Letẽ i be the error signal computed by backpropagating the synthetic gradients: Then parameters B i+1 are estimated by solving the least squares problem: Under what conditions can we show thatB i+1 → W i+1 (with enough data)? One way to find an answer is to define the synthetic gradient in terms of the system without noise added. Then B Tẽ is deterministic with respect to x, y and, assumingL has a convergent power series around ξ = 0, we can write Taken together these suggest we can proveB i+1 → W i+1 in the same way we prove consistency of the linear least squares estimator. For this to work we must show the expectation of the Taylor series approximation is well behaved. That is, we must show the expected remainder term of the expansion: is finite and goes to zero as c h → 0. This requires some additional assumptions on the problem. We make the following assumptions: • A1: the noise ξ is subgaussian, • A3: the error matricesẽ n (ẽ n) T are full rank, for 1 ≤ n ≤ N + 1, • A4: the mean of the remainder and error terms is bounded: Consider first convergence of the final layer feedback matrix, B N +1. In the final layer it is true that e N +1 = e N +1., then the least squares estimator solves and converges to the true feedback matrix, in the sense that:. We first show that, under A1-2, the conditional expectation of the estimator converges to the gradient L Taking a conditional expectation gives: We must show the remainder term goes to zero as c h → 0. This is true provided each moment E((ξ N j) m |x, y) is sufficiently well-behaved. Using Jensen's inequality and the triangle inequality in the first line, we have that With this in place, we have that the problem is close to a linear least squares problem, sincê with residual η This follows since e N +1 is defined in relation to the baseline loss, not the stochastic loss, meaning it is measurable with respect to (x, y) and can be moved into the conditional expectation. From and A3, we have that the least squares estimator satisfies Thus, using the continuous mapping theorem Then we have: lim We can use Theorem 1 to establish convergence over the rest of the layers of the network when the activation function is the identity. and σ(x) = x, the least squares estimator solves and converges to the true feedback matrix, in the sense that: Proof. DefineW n (c):= plim T →∞B n, assuming this limit exists. From Theorem 1 the top layer estimateB N +1 converges in probability toW N +1 (c). We can then use induction to establish thatB j in the remaining layers also converges in probability toW j (c). That is, assume thatB j converge in probability toW j (c) in higher layers N + 1 ≥ j > n. Then we must establish thatB n also converges in probability. To proceed it is useful to also definẽ as the error signal backpropagated through the converged (but biased) weight matricesW (c). Again it is true thatẽ N +1 = e N +1. As in Theorem 1, the least squares estimator has the form: Thus, again by the continuous mapping theorem: In this case continuity again allows us to separate convergence of each term in the product: using the weak law of large numbers in the first term, and the induction assumption for the remaining terms. In the same way Note that the induction assumption also implies limc→0ẽ n (c) = e n. Thus, putting it together, by A3, A4 and the same reasoning as in Theorem 1 we have the : solves and converges to the true feedback matrix, in the sense that: Proof. For a deep linear network notice that the node perturbation estimator can be expressed as: where the first term represents the true gradient, given by the simple linear backpropagation, the second and third terms are the remainder and a noise term, as in Theorem 1. Define Wj. Then following the same reasoning as the proof of Theorem 1, we have: Then we have: lim It is worth making the following points on each of the assumptions: • A1. In the paper we assume ξ is Gaussian. Here we prove the more general of convergence for any subgaussian random variable. • A2. In practice this may be a fairly restrictive assumption, since it precludes using relu non-linearities. Other common choices, such as hyperbolic tangent and sigmoid non-linearities with an analytic cost function do satisfy this assumption, however. • A3. It is hard to establish general conditions under whichẽ n (ẽ n) T will be full rank. While it may be a reasonable assumption in some cases. Extensions of Theorem 2 to a non-linear network may be possible. However, the method of proof used here is not immediately applicable because the continuous mapping theorem can not be applied in such a straightforward fashion as in Equation. In the non-linear case the ing sums over all observations are neither independent or identically distributed, which makes applying any law of large numbers complicated.

Represent the short summary for finding the corresponding scientific article (long)

Perturbations can be used to learn feedback weights on large fully connected and convolutional networks.

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.