Question: Represent the scientific article to find a short TLDR

Question: Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches are limited by network bandwidth, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD achieves linear speedup in the number of workers for arbitrary high compression ratios on general non-convex functions, and non-IID training data. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over decentralized user devices, connected by a peer-to-peer network and (ii) in a datacenter. Distributed machine learning-i.e. the training of machine learning models using distributed optimization algorithms-has enabled many recent successful applications in research and industry. Such methods offer two of the key success factors: 1) computational scalability by leveraging the simultaneous computational power of many devices, and 2) data-locality, the ability to perform joint training while keeping each part of the training data local to each participating device. Recent theoretical indicate that decentralized schemes can be as efficient as the centralized approaches, at least when considering convergence of training loss vs. iterations (; ; ; ;). Gradient compression techniques have been proposed for the standard distributed training case (; ; b; ;), to reduce the amount of data that has to be sent over each communication link in the network. For decentralized training of deep neural networks, introduce two algorithms (DCD, ECD) which allow for communication compression. However, both these algorithms are restrictive with respect to the used compression operators, only allowing for unbiased compressors and-more significantlyso far not supporting arbitrarily high compression ratios. We here study CHOCO-SGD-recently introduced for convex problems only -which overcomes these constraints. For the evaluation of our algorithm we in particular focus on the generalization performance (on the test-set) on standard machine learning benchmarks, hereby departing from previous work such as e.g. (; ; b;) that mostly considered training performance (on the train-set). We study two different scenarios: firstly, (i) training on a challenging peer-to-peer setting, where the training data is distributed over the training devices (and not allowed to move), similar to the federated learning setting . We are again able to show speed-ups for CHOCO-SGD over the decentralized baseline with much less communication overhead. Secondly, (ii) training in a datacenter setting, where decentralized communication patterns allow better scalability than centralized approaches. For this setting we show that communication efficient CHOCO-SGD can improve time-to-accuracy on large tasks, such as e.g. ImageNet training. However, when investigating the scaling of decentralized algorithms to larger number of nodes we observe that (all) decentralized schemes encounter difficulties and often do not reach the same (test and train) performance as centralized schemes. As these findings do point out some deficiencies of current decentralized training schemes (and are not particular to our scheme) we think that reporting these is a helpful contribution to the community to spur further research on decentralized training schemes that scale to large number of peers. • On the theory side, we are the first to show that CHOCO-SGD converges at rate O 1 / √ nT + n /(ρ 4 δ 2 T) on non-convex smooth functions, where n denotes the number of nodes, T the number of iterations, ρ the spectral gap of the mixing matrix and δ the compression ratio. The main term, O 1 / √ nT, matches with the centralized baselines with exact communication and shows a linear speedup in the number of workers n. Both ρ and δ only affect the asymptotically smaller second term. • On the practical side, we present a version of CHOCO-SGD with momentum and analyze its practical performance on two relevant scenarios: • for on-device training over a realistic peer-to-peer social network, where lowering the bandwidth requirements of joint training is especially impactful • in a datacenter setting for computational scalability of training deep learning models for resource efficiency and improved time-to-accuracy • Lastly, we systematically investigate performance of the decentralized schemes when scaling to larger number of nodes and we point out some (shared) difficulties encountered by current decentralized learning approaches. For the training in communication restricted settings a variety of methods have been proposed. For instance, decentralized schemes (; Nedić et al., 2018;), gradient compression (; ; ; ; b; ; ; b; ; ;, asynchronous methods or performing multiple local SGD steps before averaging; a). This especially covers learning over decentralized data, as extensively studied in the federated Learning literature for the centralized algorithms . In this paper we advocate for combining decentralized SGD schemes with gradient compression. Decentralized SGD. We in particular focus on approaches based on gossip averaging (; ;) whose convergence rate typically depends on the spectral gap ρ ≥ 0 of the mixing matrix . combine SGD with gossip averaging and show convergence at the rate O 1 / √ nT + n /(ρ 2 T). The leading term in the rate, O 1 / √ nT, is consistent with the convergence of the centralized mini-batch SGD and the spectral gap only affects the asymptotically smaller terms. Similar have been observed very recently for related schemes (; ;). Quantization. Communication compression with quantization has been popularized in the deep learning community by the reported successes in . Theoretical guarantees were first established for schemes with unbiased compression (; ;) but soon extended to biased compression as well. Schemes with error correction work often best in practice and give the best theoretical gurantees (b; ; ; . Recently, also proximal updates and variance reduction have been studied in combination with quantized updates Horváth et al., 2019). Decentralized Optimization with Quantization. It has been observed that gossip averaging can diverge (or not converge to the correct solution) in the presence of quantization noise (; ; Nedić et al., 2008; ; b;). propose an algorithm that can still converge, though at a slower rate than the exact scheme. Another line of work proposed adaptive schemes (with increasing compression accuracy) that converge at the expense of higher communication cost (a; ;). For deep learning applications, proposed the DCD and ECD algorithms that converge at the same rate as the centralized baseline though only for constant compression ratio. The CHOCO-SGD algorithm that we consider in this work can deal with arbitrary high compression, and has been introduced in but only been analyzed for convex functions. For non-convex functions we show a rate of, where δ > 0 measures the compression quality. Simultaneous work of Tang et al. (2019a) introduced DeepSqueeze, an alternative method which also converges with arbitrary compression ratio. In our experiments, under the same amount of tuning, CHOCO-SGD achieves higher test accuracy. Algorithm 1 CHOCO-SGD input:, E) and mixing matrix W, initializex 1: for t in 0... T − 1 do {in parallel for all workers i ∈ [n]} 2: for neighbors j: {i, j} ∈ E (including {i} ∈ E) do 5: end for 8: 9: In this section we formally introduce the decentralized optimization problem, compression operators, and the gossip-based stochastic optimization algorithm CHOCO-SGD from . Distributed Setup. We consider optimization problems distributed across n nodes of the form where D 1,... D n are local distributions for sampling data which can be different on every node, Communication. Every device is only allowed to communicate with its local neighbours defined by the network topology, given as a weighted graph G = ([n], E), with edges E representing the communication links along which messages (e.g. model updates) can be exchanged. We assign a positive weight w ij to every edge (w ij = 0 for disconnected nodes {i, j} / ∈ E). Assumption 1 (Mixing matrix). We assume that In our experiments we set the weights based on the local node degrees: w ij = max{deg(i), deg(j)} −1 for {i, j} ∈ E. This will not only guarantee ρ > 0 but these weights can easily be computed in a local fashion on each node . Compression. We aim to only transmit compressed (e.g. quantized or sparsified) messages. We formalized this through the notion of compression operators that was e.g. also used in . for a parameter δ > 0. Here E Q denotes the expectation over the internal randomness of operator Q. In contrast to the quantization operators used in e.g. (; Horváth et al., 2019), compression operators defined as in are not required to be unbiased and therefore supports a larger class of compression operators. Some examples can be found in and we further discuss specific compression schemes in Section 5. Algorithm. CHOCO-SGD is summarized in Algorithm 1. Every worker i stores its own private variable x i ∈ R d that is updated by a stochastic gradient step in part 2 and a modified gossip averaging step on line 2. This step is a key element of the algorithm as it preserves the averages of the iterates even in presence of quantization noise (the compression errors are not discarded, but aggregated in the local variables x i, see also ). The nodes communicate with their neighbors in part 1 and update the variablesx j ∈ R d for all their neighbors {i, j} ∈ E only using compressed updates. Thesex i are available to all the neighbours of the node i and represent the'publicly available' copies of the private x i, in general x i =x i, due to the communication restrictions. From an implementation aspect, it is worth highlighting that the communication part 1 and the gradient computation part 2 can both be executed in parallel because they are independent. Moreover, each node only needs to store 3 vectors at most, independent of the number of neighbors (this might not be obvious from the notation used here for additinal clarity, for further details c.f. ). We further propose a momentum-version of CHOCO-SGD in Algorithm 2 (see also Section D for further details). As the first main contribution, we here extend the analysis of CHOCO-SGD to non-convex problems. For this we make the following technical assumptions: and the variance of the stochastic gradients is bounded on each worker: where 2, the averaged iterates i of Algorithm 1 satisfy: where c:= ρ 2 δ 82 denotes the convergence rate of the underlying consensus averaging scheme of . This shows that CHOCO-SGD converges asymptotically as The first term shows a linear speed-up compared to SGD on a single node, while compression and graph topology affect only the higher order second term. For slightly more general statements than Theorem 4.1 (with improved constants) as well as for the proofs and convergence of the individual iterates x i we refer to Appendix A. In this section we experimentally compare CHOCO-SGD to the relevant baselines for a selection of commonly used compression operators. For the experiments we further leverage momentum in all implemented algorithms. The newly developed momentum version of CHOCO-SGD is given as Algorithm 2. Algorithm 2 CHOCO-SGD with Momentum input: Same as for Algorithm 1, additionally: weight decay factor λ, momentum factor β, local momentum memory v i:= 0 ∀i ∈ [n] Lines 1-8 in Algorithm 1 are left unmodified Line 9 in Algorithm 1 is replaced with the following two lines local momentum with weight decay 10: Setup. In order to match the setting in for our first set of experiments, we use a ring topology with n = 8 nodes and train the ResNet20 architecture on the Cifar10 dataset (50K/10K training/test samples) . We randomly split the training data between workers and shuffle it after every epoch, following standard procedure as e.g. in . We implement DCD and ECD with momentum , DeepSqueeze with momentum (a), CHOCO-SGD with momentum (Algorithm 2) and standard (all-reduce) mini-batch SGD with momentum and without compression . The momentum factor is set to 0.9 without dampening. For all algorithms we fine-tune the initial learning rate and gradually warm it up from a relative small value (0.1) for the first 5 epochs. The learning rate is decayed by 10 twice, at 150 and 225 epochs, and stop training at 300 epochs. For CHOCO-SGD and DeepSqueeze the consensus learning rate γ is also tuned. The detailed hyper-parameter tuning procedure refers to Appendix F. Every compression scheme is applied to every layer of ResNet20 separately. We evaluate the top-1 test accuracy on every node separately over the whole dataset and report the average performance over all nodes. Compression Schemes. We implement two unbiased compression schemes: (i) gsgd b quantization that randomly rounds the weights to b-bit representations , and (ii) random a sparsification, which preserves a randomly chosen a fraction of the weights and sets the other ones to zero . Further two biased compression schemes: (iii) top a, which selects the a fraction of weights with the largest magnitude and sets the other ones to zero , and (iv) sign compression, which compresses each weight to its sign scaled by the norm of the full vector (; . We refer to Appendix C for exact definitions of the schemes. DCD and ECD have been analyzed only for unbiased quantization schemes, thus the combination with the two biased schemes is not supported by theory. In converse, CHOCO-SGD and DeepSqueeze has been studied only for biased schemes according to Definition 2. However, both unbiased compression schemes can be scaled down in order to meet the specification (cf. discussions in ) and we adopt this for the experiments. Results. The are summarized in Table 1. For unbiased compression schemes, ECD and DCD only achieve good performance when the compression ratio is small, and sometimes even diverge when the compression ratio is high. This is consistent 1 with the theoretical and experimental in . We further observe that the performance of DCD with the biased top a sparsification is much better than with the unbiased random a counterpart, though this operator is not yet supported by theory. CHOCO-SGD can generalize reasonably well in all scenarios (at most 1.65% accuracy drop) for fixed training budget. The sign compression achieves state-of-the-art accuracy and requires approximately 32× less bits per weight than the full precision baseline. We now shift our focus to challenging real-world scenarios which are intrinsically decentralized, i.e. each part of the training data remains local to each device, and thus centralized methods either fail or are inefficient to implement. Typical scenarios comprise e.g. sensor networks, or mobile devices or hospitals which jointly train a machine learning model. Common to these applications is that i) each device has only access to locally stored or acquired data, ii) communication bandwidth is limited (either physically, or artificially for e.g. metered connections), iii) the global network topology is typically unknown to a single device, and iv) the number of connected devices is typically large. Additionally, this fully decentralized setting is also strongly motivated by privacy aspects, enabling to keep the training data private on each device at all times. Modeling. To simulate this scenario, we permanently split the training data between the nodes, i.e. the data is never shuffled between workers during training, and every node has distinct part of the dataset. To the best of our knowledge, no prior works studied this scenario for decentralized Figure 1: Scaling of CHOCO-SGD with sign compression to large number of devices on Cifar10 dataset. Left: best testing accuracy of the algorithms reached after 300 epochs. Right: best testing accuracy reached after communicating 1000 MB. deep learning. For the centralized approach, gathering methods such as all-reduce are not efficiently implementable in this setting, hence we compare to the centralized baseline where all nodes route their updates to a central coordinator for aggregation. For the comparison we consider CHOCO-SGD with sign compression (this combination achieved the compromise between accuracy and compression level in Table 1)), decentralized SGD without compression, and centralized SGD without compression. Scaling to Large Number of Nodes. To study the scaling properties of CHOCO-SGD, we train on 4, 16, 36 and 64 number of nodes. We compare decentralized algorithms on two different topologies: ring as the worst possible topology, and on the torus with much larger spectral gap. Their parameters are listed in the table 2. . For the simplicity, we keep the learning rate constant and separately tune it for all methods. We tune consensus learning rate for CHOCO-SGD. The are summarized in Figure 1. First we compare the testing accuracy reached after 300 epochs (Fig. 1, Left). CentralizedSGD has a good performance for all the considered number of nodes. CHOCO-SGD slows down due to the influence of graph topology (Decentralized curve), which is consistent with the spectral gaps order (see Tab. 2), and also influenced by the communication compression (CHOCO curve), which slows down training uniformly for both topologies. We observed that the train performance is similar to the test on Fig. 1, therefore the performance degradation is explained by the slower convergence (Theorem 4.1) and is not a generalization issue. Increasing the number of epochs improves the performance of the decentralized schemes. However, even using 10 times more epochs, we were not able to perfectly close the gap between centralized and decentralized algorithms for both train and test performance. In the real decentralized scenario, the interest is not to minimize the epochs number, but the amount of communication to reduce the cost of the user's mobile data. We therefore fix the number of transmitted bits to 1000 MB and compare the best testing accuracy reached (Fig. 1, Right). CHOCO-SGD performs the best while having slight degradation due to increasing number of nodes. It is beneficial to use torus topology when the number of nodes is large because it has good mixing properties, for small networks there is not much difference between these two topologies-the benefit of large spectral gap is canceled by the increased communication due larger node degree for torus topology. Both Decentralized and Centralized SGD requires significantly larger number of bits to reach reasonable accuracy. Experiments on a Real Social Network Graph. We simulate training models on user devices (e.g. mobile phones), connected by a real social network. We chosen Davis Southern women social network with 32 nodes. We train ResNet20 (0.27 million parameters) model on the Cifar10 dataset (50K/10K training/test samples) for image classification and a three-layer LSTM architecture (28.95 million parameters) for a language modeling task on WikiText-2 (600 training and 60 validation articles with a total of 2 088 628 and 217 646 tokens respectively) . We use exponentially decaying learning rate schedule. For more detailed experimental setup we refer to Appendix F. The are summarized in Figures 2-3 and in Table 3. For the image classification task, when comparing the training accuracy reached after the same number of epochs, we observe that the decentralized algorithm performs best, follows by the centralized and lastly the quantized decentralized. However, the test accuracy is highest for the centralized scheme. When comparing the test accuracy reached for the same transmitted data 2, CHOCO-SGD significantly outperforms the exact decentralized scheme, with the centralized performing worst. We note a slight accuracy drop, i.e. after the same number of epochs (but much less transmitted data), CHOCO-SGD does not reach the same level of test accuracy than the baselines. For the language modeling task, both decentralized schemes suffer a drop in the training loss when the evaluation reaching the epoch budget; while our CHOCO-SGD outperforms the centralized SGD in test perplexity. When considering perplexity for a fixed data volume (middle and right subfigure of Figure 3), CHOCO-SGD performs best, followed by the exact decentralized and centralized algorithms. Figure 4: Large-scale training: Resnet-50 on ImageNet-1k in the datacenter setting. The topology has 8 nodes (each accesses 4 GPUs). We use "Sign+Norm" as the quantization scheme of CHOCO-SGD. The benefits of CHOCO-SGD can be further pronounced when scaling to more nodes. Decentralized optimization methods offer a way to address scaling issues even for well connected devices, such as e.g. in datacenter with fast InfiniBand (100Gbps) or Ethernet (10Gbps) connections. describe scenarios when decentralized schemes can outperform centralized ones, and recently, presented impressive speedups for training on 256 GPUs, for the setting when all nodes can access all training data. The main differences of their algorithm to CHOCO-SGD are the asynchronous gossip updates, time-varying communication topology and most importantly exact communication, making their setup not directly comparable to ours. We note that these properties of asynchronous communication and changing topology for faster mixing are orthogonal to our contribution, and offer promise to be combined. Setup. We train ImageNet-1k (1.28M/50K training/validation) with Resnet-50 . We perform our experiments on 8 machines (n1-standard-32 from Google Cloud), where each of machines has 4 Tesla P100 GPUs. Within one machine communication is fast and we perform all-reduce with the full model. Between different machines we use decentralized communication with compressed communication (sign-CHOCO-SGD) in a ring topology. The mini-batch size on each GPU is 128, and we follow the general SGD training scheme in and directly use all their hyperparameters for CHOCO-SGD. Due to the limitation of the computational resource, we did not heavily tune the consensus stepsize for CHOCO-SGD 3. Results. We depict the training loss and top-1 test accuracy in terms of epochs and time in Figure 4. CHOCO-SGD benefits from its decentralized and parallel structure and takes less time than all-reduce to perform the same number of epochs, while having only a slight 1.5% accuracy loss. (All-reduce with full precision gradients achieved test accuracy of 76.37%, vs. 75.15% for CHOCO-SGD). In terms of time per epoch, our speedup does not match that of , as the used hardware is very different. Their scheme is orthogonal to our approach and could be integrated for better training efficiency. Nevertheless, we still demonstrate a time-wise 20% gain over the common all-reduce baseline, on our used commodity hardware cluster. We propose the use of CHOCO-SGD (and its momentum version) for enabling decentralized deep learning training in bandwidth-constrained environments. We provide theoretical convergence guarantees for the non-convex setting and show that the algorithm enjoys the a linear speedup in the number of nodes. We empirically study the performance of the algorithm in a variety of settings on image classification (ImageNet-1k, Cifar10) and on a language modeling task (WikiText-2). Whilst previous work successfully demonstrated that decentralized methods can be a competitive alternative to centralized training schemes when no communication constraints are present , our main contribution is to enable training in strongly communication-restricted environments, and while respecting the challenging constraint of locality of the training data. We theoretically and practically demonstrate the performance of decentralized schemes for arbitrary high communication compression, and under data-locality, and thus significantly expand the reach of potential applications of fully decentralized deep learning. In this section we present the proof of Theorem 4.1. For this, we will first derive a slightly more general statement: in Theorem A.2 we analyze CHOCO-SGD for arbitrary stepsizes η, and then derive Theorem 4.1 as a special case. The structure of the proof follows. That is, we first show that Algorithm 1 is a special case of a more general class of algorithms (given in Algorithm 3): Observe that Algorithm 1 consists of two main components: 2 the stochastic gradient update, performed locally on each node, and 1 the (quantized) averaging among the nodes. We can show convergence of all algorithms of this type-i.e. stochastic gradient updates 2 followed by an arbitrary averaging step 1 -as long as the averaging scheme exhibits linear convergence. For the specific averaging used in CHOCO-SGD, linear convergence has been shown in and we will use their estimate of the convergence rate of the averaging scheme. For convenience, we use the following matrix notation in this subsection. Decentralized SGD with arbitrary averaging is given in Algorithm 3. Algorithm 3 DECENTRALIZED SGD WITH ARBITRARY AVERAGING SCHEME input: blackbox averaging/gossip 4: end for 2 {1 { Assumption 3. For an averaging scheme h : Assume that h preserves the average of iterates: and that it converges with linear rate for a parameter 0 < c ≤ 1 and Laypunov function Ψ(X,, where E h denotes the expectation over internal randomness of averaging scheme h. Example: Exact Averaging. Setting X + = XW and Y + = X + gives an exact consensus averaging algorithm with mixing matrix W . It converges at the rate c = ρ, where ρ is an eigengap of mixing matrix W, defined in Assumption 1. Substituting it into the Algorithm 3 we recover D-PSGD algorithm, analyzed in. CHOCO-SGD. To recover CHOCO-SGD, we need to choose CHOCO-GOSSIP as consensus averaging scheme, which is defined as 82 in the more general below. It is important to note that for Algorithm 1 given in the main text, the order of the communication part 1 and the gradient computation part 2 is exchanged. We did this to better illustrate that both these parts are independent and that they can be executed in parallel. The effect of this change can be captured by changing the initial values but does not affect the convergence rate. A.2 PROOF OF THEOREM 4.1 Lemma A.1. Under Assumptions 1-2 the iterates of the Algorithm 3 with constant stepsize η satisfy Proof of Lemma A.1. These are exactly the same calculations as the first 9 lines in the proof of Lemma 21 from. We got a recursion Verifying that r t ≤ η 2 4A c 2 satisfy recursion completes the proof as E X Theorem A.2. Under Assumptions 1-3 with constant stepsize η = n T +1 for T ≥ 64nL 2, the averaged iterates i of Algorithm 3 satisfy: where c denotes convergence rate of underlying averaging scheme. The first term shows a linear speed up compared to SGD on one node, whereas the underlying averaging scheme affects only the second-order term. Substituting the convergence rate for exact averaging with W gives the rate O(1 / √ nT + n /(T ρ 2)), which recovers the rate of D-PSGD . CHOCO-SGD with the underlying CHOCO-GOSSIP averaging scheme converges at the rate 2 )). The dependence on ρ (eigengap of the mixing matrix W) is worse than in the exact case. This might either just be an artifact of our proof technique or a consequence of supporting arbitrary high compression. Proof of Theorem A.2. By L-smoothness To estimate the second term, we add and subtract ∇f (For the last term, we add and subtract ∇f (x (t) ) and the sum of ∇f j (x Combining this together and using L-smoothness to estimate f ( Using Lemma A.1 to bound the third term and using that η ≤ Rearranging terms and averaging over t Substituting η = n T +1 and using that T ≥ 64nL 2 we get the statement of the theorem. The theorem gives guarantees for the averaged vector of parameters x, however in a decentralized setting it is very expensive and sometimes impossible to average all the parameters distributed across several machines, especially when the number of machines and the model size is large. We can get similar guarantees on the individual iterates x i as e.g. in . We summarize these briefly below. Corollary A.3 (Convergence of local weights). Under the same setting as in Theorem 4.1, Proof of Corollary A.3. where we used L-smoothness of f. Using Theorem 4.1 and Lemma A.1 The previous holds only for T larger than 64nL 2. This is not necessary and can be relaxed. Theorem A.4. Under Assumptions 2, 1 with constant stepsize η and the consensus stepsize γ:= ρ 2 δ 16ρ+ρ 2 +4β 2 +2ρβ 2 −8ρδ where β = I − W 2 ∈ Algorithm 3 converges at the speed where i and c is convergence rate of underlying averaging scheme. In contrast to Theorem A.2, this rate holds for any T, however the first term is worse than in Theorem A.2 because σ 2 is usually much smaller than G 2. Proof of Theorem A.4. By L-smoothness Using Lemma A.1 we can bound the last term Rearranging terms, taking α = 1 and averaging over t we are getting statement of the theorem Corollary A.5 (Convergence of local weights x (t) i ). Under Assumtions 2, 1, algorithm 1 with η = n T +1 converges at the speed c 2 (T + 1). Proof. where we used L-smoothness of f. This holds for ∀α > 0. Using Theorem A.4 and Lemma A.1 and setting α = 1 where we set η = n T +1. Lemma B.1. For arbitrary set of n vectors Lemma B.3. For given two vectors a, This inequality also holds for the sum of two matrices A, B ∈ R n×d in Frobenius norm. Algorithm 4 CHOCO-SGD as Error Feedback, E) and mixing matrix W, initializex 1: for t in 0... T − 1 do {in parallel for all workers i ∈ [n]} 2: for neighbors j: {i, j} ∈ E (including {i} ∈ E) do end for 10: 11: stochastic gradient update 12: end for D CHOCO-SGD WITH MOMENTUM Algorithm 2 demonstrates how to combine CHOCO-SGD with weight decay and momentum. Nesterov momentum can be analogously adapted for our decentralized setting. To better understand how does CHOCO-SGD work, we can interpret it as an error feedback algorithm (; . We can equivalently rewrite CHOCO-SGD (Algorithm 1) as Algorithm 4. The common feature of error feedback algorithms is that quantization errors are saved into the internal memory, which is added to the compressed value at the next iteration. In CHOCO-SGD the value we want to transmit is the difference x (t) i − x (t−1) i, which represents the evolution of local variable x i at step t. Before compressing this value on line 4, the internal memory is added on line 3 to correct for the errors. Then, on line 5 internal memory is updated. Note that m i in the old notation. We precise the procedure of model training as well as the hyper-parameter tuning in this section. Social Network Setup. For the comparison we consider CHOCO-SGD with sign compression (this combination achieved the compromise between accuracy and compression level in Table 1)), decentralized SGD without compression, and centralized SGD without compression. We train two models, firstly ResNet20 (0.27 million parameters) for image classification on the Cifar10 dataset (50K/10K training/test samples) and secondly, a three-layer LSTM architecture (28.95 million parameters) for a language modeling task on WikiText-2 (600 training and 60 validation articles with a total of 2 088 628 and 217 646 tokens respectively) . For the language modeling task, we borrowed and adapted the general experimental setup of , where we use a three-layer LSTM with hidden dimension of size 650. The loss is averaged over all examples and timesteps. The BPTT length is set to 30. We fine-tune the value of gradient clipping (0.4), and the dropout (0.4) is only applied on the output of LSTM. We train both of ResNet20 and LSTM for 300 epochs, unless mentioned specifically. The per node mini-batch size is 32 for both datasets. The learning rate of CHOCO-SGD follows a linear scaling rule, which is proportional to the node degree. The momentum (with factor 0.9) is only applied on the ResNet20 training. Social Network and a Datacenter details. For all algorithms, we gradually warmup the learning rate from a relative small value (0.1) to the fine-tuned initial learning rate for the first 5 training epochs. During the training procedure, the tuned initial learning rate is decayed by the factor of 10 when accessing 50% and 75% of the total training epochs. The learning rate is tuned by finding the optimal learning rate per sampleη, where the learning rate (used locally) is determined by a linear scaling rule (i.e., degree of node ×η × per node mini-batch size). The optimalη is searched in a pre-defined grid and we ensure that the best performance was contained in the middle of the grids. For example, if the best performance was ever at one of the extremes of the grid, we would try new grid points. Same searching logic applies to the consensus stepsize. Table 4 demonstrates the fine-tuned hpyerparameters of CHOCO-SGD for training ResNet-20 on Cifar10, while Table 6 reports our fine-tuned hpyerparameters of our baselines. Table 5 demonstrates the fine-tuned hpyerparameters of CHOCO-SGD for training ResNet-20/LSTM on a social network topology. Table 5: Tuned hyper-parameters of CHOCO-SGD, corresponding to the social network topology with 32 nodes in Table 3. We randomly split the training data between the nodes and keep this partition fixed during the entire training (no shuffling). The per node mini-batch size is 32 and the maximum degree of the node is 14. Base learning rate (before scaling by node degree) Consensus stepsize Figure 1. n = 4 n = 16 n = 36 n = 64 Figure 1. n = 4 n = 16 n = 36 n = 64 We additionally plot the learning curve for the social network topology in Figure. 6 and Figure. topology. The topology has 32 nodes and we assume each node can only access a disjoint subset of the whole dataset. The local mini-batch size is 32. We additionally provide plots for training top-1, top-5 accuracy and test top-5 accuracy for the datacenter experiment in Figure 8. On Figure 9 we additionally depict the test accuracy of the averaged model i (left) and averaged distance of the local models from the averaged model (right). Towards the end of the optimization the local models reach consensus (Figure 9, right), and their individual test performances are the same as performance of averaged model. Interestingly, before decreasing the stepsize at the epoch 225, the local models are in general diverging from the averaged model, while decreasing only when the stepsize decreases. The same behavior was also reported in.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Choco-SGD---decentralized SGD with compressed communication---for non-convex objectives and show its strong performance in various deep learning applications (on-device learning, datacenter case).

Represent the short summary for finding the corresponding scientific article (long)

We investigate the convergence of popular optimization algorithms like Adam , RMSProp and propose new variants of these methods which provably converge to optimal solution in convex  settings.


Question: Represent the article for finding a single-sentence TLDR

Question: We show that Entropy-SGD , when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound’s prior, violating the hypothesis of the PAC-Bayes theorem that the prior is chosen independently of the data. Indeed, available implementations of Entropy-SGD rapidly obtain zero training error on random labels and the same holds of the Gibbs posterior. In order to obtain a valid generalization bound, we show that an ε-differentially private prior yields a valid PAC-Bayes bound, a straightforward consequence of connecting generalization with differential privacy. Using stochastic gradient Langevin dynamics (SGLD) to approximate the well-known exponential release mechanism, we observe that generalization error on MNIST (measured on held out data) falls within the (empirically nonvacuous) bounds computed under the assumption that SGLD produces perfect samples. In particular, Entropy-SGLD can be configured to yield relatively tight generalization bounds and still fit real labels, although these same settings do not obtain state-of-the-art performance. Optimization is central to much of machine learning, but generalization is the ultimate goal. Despite this, the generalization properties of many optimization-based learning algorithms are poorly understood. The standard example is stochastic gradient descent (SGD), one of the workhorses of deep learning, which has good generalization performance in many settings, even under overparametrization BID36, but rapidly overfits in others BID44. Can we develop high performance learning algorithms with provably strong generalization guarantees? Or is their a limit?In this work, we study an optimization algorithm called Entropy-SGD BID10, which was designed to outperform SGD in terms of generalization error when optimizing an empirical risk. Entropy-SGD minimizes an objective f: R p → R indirectly by performing (approximate) stochastic gradient ascent on the so-called local entropy DISPLAYFORM0 where C is a constant and N denotes a zero-mean isotropic multivariate normal distribution on R p.Our first contribution is connecting Entropy-SGD to in statistical learning theory, showing that maximizing the local entropy corresponds to minimizing a PAC-Bayes bound BID31 on the risk of the so-called Gibbs posterior. The distribution of w + ξ is the PAC-Bayesian "prior", and so optimizing the local entropy optimizes the bound's prior. This connection between local entropy and PAC-Bayes follows from a due to Catoni (2007, Lem. 1.1.3) in the case of bounded risk. (See Theorem 4.1.) In the special case where f is the empirical cross entropy, the local entropy is literally a Bayesian log marginal density. The connection between minimizing PACBayes bounds under log loss and maximizing log marginal densities is the subject of recent work by BID19. Similar connections have been made by BID45; Zhang (2006b); BID20; BID21.Despite the connection to PAC-Bayes, as well as theoretical by Chaudhari et al. suggesting that Entropy-SGD may be more stable than SGD, we demonstrate that Entropy-SGD (and its corresponding Gibbs posterior) can rapidly overfit, just like SGD. We identify two changes, motivated by theoretical analysis, that suffice to control generalization error, and thus prevent overfitting. The first change relates to the stability of optimizing the prior mean. The PAC-Bayes theorem requires that the prior be independent of the data, and so by optimizing the prior mean, Entropy-SGD invalidates the bound. Indeed, the bound does not hold empirically. While a PAC-Bayes prior may not be chosen based on the data, it can depend on the data distribution. This suggests that if the prior depends only weakly on the data, it may be possible to derive a valid bound. We formalize this intuition using differential privacy BID13 BID17. By modifying the cross entropy loss to be bounded and replacing SGD with stochastic gradient Langevin dynamics (SGLD; BID43, the data-dependent prior mean can be shown to be (ε, δ)-differentially private BID42 BID34. We refer to the SGLD variant as Entropy-SGLD. Using connecting statistical validity and differential privacy (b, Thm. 11), we show that an ε-differentially private prior mean yields a valid, though looser, generalization bound using the PAC-Bayes theorem. (See Theorem 5.4.)A gap remains between pure and approximate differential privacy. Under some technical conditions, in the limit as the number of iterations diverges, the distribution of SGLD's output is known to converge weakly to the corresponding stationary distribution, which is the well-known exponential mechanism in differential privacy (, Thm. 7). Weak convergence, however, falls short of implying that SGLD achieves pure ε-differential privacy. We proceed under the approximation that SGLD enjoys the same privacy as the exponential release mechanism, and apply our ε-differentially private PAC-Bayes bound. We find that the corresponding 95% confidence intervals are reasonably tight but still conservative in our experiments. While the validity of our bounds are subject to our approximation, the bounds give us a view as to the limitations of combining differential privacy with PAC-Bayes bounds: when the privacy of Entropy-SGLD is tuned to contribute no more than 2ε 2 × 100 ≈ 0.2% to the generalization error, the test error of the learned network is 3-8%, which is approximately 5-10 times higher than the state of the art, which for MNIST is between 0.2-1%, although the community has almost certainly overfit its networks/learning rates/loss functions/optimizers to MNIST. We return to these points in the discussion. The second change pertains to the stability of the stochastic gradient estimate made on each iteration of Entropy-SGD. This estimate is made using SGLD. (Hence Entropy-SGD is SGLD within SGD.) Chaudhari et al. make a subtle but critical modification to the noise term in SGLD update: the noise is divided by a factor that ranges from 10 3 to 10 4. (This factor was ostensibly tuned to produce good empirical .) Our analysis shows that, as a of this modification, the Lipschitz constant of the objective function is approximately 10 6 -10 8 times larger, and the that the Entropy-SGD objective is smoother than the original risk surface no longer stands. This change to the noise also negatively impacts the differential privacy of the prior mean. Working backwards from the desire to obtain tight generalization bounds, we are led to divide the SGLD noise by a factor of only 4 √ m, where m is the number of data points. (For MNIST, 4 √ m ≈ 16.) The ing bounds are nonvacuous and tighter than those recently published by BID18, although it must be emphasized that the bound presented here hold subject to the approximation concerning privacy of the prior mean, which is certainly violated but to an unknown degree. We begin with a review of some related work, before introducing sufficient so that we can make a formal connection between local entropy and PAC-Bayes bounds. We then introduce a differentially private PAC-Bayes bound. In Section 6, we present experiments on MNIST which provide evidence for our theoretical analysis. (Empirical validation is required in order to address the aforementioned gap between pure and approximate differential privacy.) We close with a short discussion. This work was inspired in part by BID44, who highlight empirical properties of SGD that were not widely appreciated within the theory community, and propose a simple linear model to explain the phenomenon. They observe that, without regularization, SGD can achieve zero training error on MNIST and CIFAR, even if the labels are chosen uniformly at random. At the same time, SGD obtains weights with very small generalization error with the original labels. The first observation is strong evidence that the set of classifier accessible to SGD within a reasonable number of iterations is extremely rich. Indeed, with probability almost indistinguishable from one, fitting random labels on a large data set implies that the Rademacher complexity of this effective hypothesis class is essentially the maximum possible (, Thm. 11).The second observation suggests that SGD is performing some sort of capacity control. Zhang et al. show that SGD obtains the minimum norm solution for a linear model, and thus performs implicit regularization. They suggest a similar phenomenon may occur when using SGD to training neural networks. Indeed, earlier work by observed similar phenomena and argued for the same point: implicit regularization underlies the ability of SGD to generalize, even under massive overparametrization. Subsequent work by introduced "path" norms as a better measure of the complexity of ReLU networks. Despite progress, these new norms have not yet lead to nonvacuous generalization bounds (, App. D).There has been recent progress: describe PAC-Bayes bounds, built by perturbing the weights learned by SGD. (The authors were motivated in part by Entropy-SGD and empirical findings relating to "flat minima".) Their bounds are controlled by 1) the "flatness" of empirical risk surface near the SGD solution and 2) the L2 distance between the learned weights and the random initialization. The bounds are also found to be numerically nonvacuous. (We return to this aspect below.) Similar bounds are studied in further depth by Neyshabur et al. (2017b). Recent advances have also identified new spectral norm bounds that correlate closely with generalization error and distinguish between true and random labels BID4 BID38.Our work and Entropy-SGD both connect to early work by BID22 and BID23, which introduced regularization schemes based on information-theoretic principles. These ideas, now referred to as "flat minima", were related to minimizing PAC-Bayes bounds by BID18, although these bounds are minimized with respect to the posterior, not the prior, as is done by Entropy-SGD. BID1 provide an informationtheoretic argument for a generalization of the objective of Hinton and Camp. Their objective takes the form of regularized empirical cross entropŷ DISPLAYFORM0 where Q and P are the prior and posterior on the weights, respectively. For an appropriate range of β, linear PAC-Bayes bounds are exactly of this form. they empirically observe that varying β correlates with a degree of overfitting on a random label dataset. BID1 also highlight the connections with variational inference BID25.Our work also relates to renewed interest in nonvacuous generalization bounds BID26 BID27, i.e., bounds on the numerical difference between the unknown classification error and the training error that are (much) tighter than the tautological upper bound of one. demonstrated nonvacuous generalization bounds for random perturbations of SGD solutions using PAC-Bayes bounds for networks with millions of weights.(The algorithm can be viewed as variational dropout BID25, with a proper data-dependent prior but without local reparametrization.) Their work builds on the core insight demonstrated nearly 15 years ago by BID27, who computed nonvacuous bounds for neural networks five orders of magnitude smaller. A key aspect of our analysis relies on the stability of a data-dependent prior. Stability has long been understood to relate to generalization BID8. Our analysis of Entropy-SGLD rests on in differential privacy (see for a survey) and its connection to generalization BID17 BID16 BID7 BID40, which can be viewed as a particularly stringent notion of stability. Entropy-SGLD is an instance of differentially private empirical risk minimization, which is well studied, both in the abstract BID11 BID24 BID6 and in the particular setting of private training via SGD BID6 BID0. Our analysis also relates to the differential privacy of Bayesian and Gibbs posteriors, and approximate sampling algorithms BID35 BID6 BID12 BID42 BID34.In effect, our differentially private PAC-Bayes bound uses a data-distribution-dependent prior, which are permitted in the PAC-Bayesian framework. (Priors must be independent of the data sample, however. Differential privacy allows us to extract information about the distribution from a sample while maintaining statistical validity BID17 .)There is a growing body of work in the PAC-Bayes literature on data-distribution-dependent priors. Write S for a data sample and Q(S) for a data-dependent PAC-Bayesian posterior (i.e., Q : Z m → M 1 (R p) is a fixed learning algorithm for a randomized classifier). BID9 makes an extensive study of data-distribution-dependent priors of the form P * = P * (Q) DISPLAYFORM1. While such priors were known to minimize the KL term in expectation, Catoni was the first to derive PAC-Bayes excess risk bounds using these priors: focusing on Gibbs posteriors Q(S) = Q P (S) def = P exp(−τR S) for some fixed measure P, Catoni derives bounds on the complexity term KL(Q P (S)||P * (Q P)) that hold uniformly over all possible data distributions D. Catoni calls such priors and bounds "local". BID30 extend this approach to generalization bounds and consider both data-independent and data-dependent choices for P. In the later case, P = P(S) and the generalization bound uses the local prior P * (Q P) = E S∼D m [Q P(S) (S)]. In our work, we make a data-dependent but private choice of the prior P = P(S), and then use our differentially private PAC-Bayes generalization bound to control the generalization error of the associated Gibbs posterior Q P (S) in terms of KL(Q P (S)||P). We also evaluated differentially private versions of local bounds, where the complexity term is a uniform bound on KL(Q P (S)||P * (Q P)). The bounds were virtually indistinguishable, and so we do not report them here.3 PRELIMINARIES: SUPERVISED LEARNING, ENTROPY-SGD, AND PAC-BAYES Let Z be a measurable space, let D be an unknown distribution on Z, and consider the batch supervised learning setting under a loss function bounded below: having observed S ∼ D m, i.e., m independent and identically distributed samples from D, we aim to choose a predictor, parameterized by weight vector w ∈ R p, with minimal risk DISPLAYFORM2 where: R p × Z → R is measurable and bounded below. (We ignore the possibility of constraints on the weight vector for simplicity.) We will also consider randomized predictors, represented by probability measures Q ∈ M 1 (R p) on R p, whose risks are defined via averaging, DISPLAYFORM3 where the second equality follows from Fubini's theorem and the fact that is bounded below. Let S = (z 1, . . ., z m) and letD DISPLAYFORM4 δ z i be the empirical distribution. Given a weight distribution Q, such as that chosen by a learning algorithm on the basis of data S, its empirical risk DISPLAYFORM5 will be studied as a stand-in for its risk, which we cannot compute. WhileR S (Q) is easily seen to be an unbiased estimate of R D (Q) when Q is independent of S, our goal is to characterize the (one-sided) generalization error R D (Q) −R S (Q) when Q is random and dependent on S.One of our focuses will be on classification, where Z = X × K, with K a finite set of classes/labels. A product measurable (in practice, continuous) function f: DISPLAYFORM6 In this setting, 0-1 loss corresponds to g(y, y) = 1 if and only if y = y. In binary classification, we take K = {0, 1}.We will also consider parametric families of probability-distribution-valued classifiers f: DISPLAYFORM7 For every input x ∈ X, the output f (w, x) specifies a probability distribution on K. In this setting, (w, (x, y)) = g(f (w, x), y) for some g: DISPLAYFORM8 The standard loss is then the cross entropy, given by g((p 1, . . ., p K), y) = − log p y. (Under cross entropy loss, the empirical risk is, up to a multiplicative constant, a negative log likelihood.) In the special case of binary classification, the output can be represented simply by an element of, i.e., the probability the label is one. The binary cross entropy, BCE, is given by g(p, y) = −y log(p) − (1 − y) log(1 − p). Note that cross entropy loss is merely bounded below. We will consider bounded modifications in Appendix B.2.We will sometimes refer to elements of R p and M 1 (R p) as classifiers and randomized classifiers, respectively. Likewise, we will often refer to the (empirical) risk as the (empirical) error. Entropy-SGD is a gradient-based learning algorithm proposed by BID10 as an alternative to stochastic gradient descent on the empirical risk surfaceR S. The authors argue that Entropy-SGD has better generalization performance and provide some empirical evidence. Part of that argument is a theoretical analysis of the smoothness of the local entropy surface that Entropy-SGD optimizes in place of the empirical risk surface, as well as a uniform stability argument that they admit rests on assumptions that are violated, but to a small degree empirically. As we have mentioned in the introduction, Entropy-SGD's modifications to the noise term in SGLD in much worse smoothness. We will modify Entropy-SGD in order to stabilize its learning and, up to some approximations, provably control overfitting. Entropy-SGD is stochastic gradient ascent applied to the optimization problem:arg max DISPLAYFORM0 The objective F γ,τ (·; S) is known as the local entropy, and can be viewed as the log partition function of the unnormalized probability density function DISPLAYFORM1 (We will denote the corresponding distribution by G w,S γ,τ .) Assuming that one can exchange differentiation and integration, it is straightforward to verify that DISPLAYFORM2 and then the local entropy F γ,τ (·; S) is even differentiable, even if the empirical riskR S is not. Indeed, Chaudhari et al. show that the local entropy and its derivative are Lipschitz. Chaudhari et al. argue informally that maximizing the local entropy leads to "flat minima" in the empirical risk surface, which several authors BID22 BID23 BID2 BID3 have argued is tied to good generalization performance (though none of these papers gives generalization bounds, vacuous or otherwise). 1 Chaudhari et al. propose a Monte Carlo estimate of the gradient, DISPLAYFORM3 1 The local entropy should not be confused with the smoothed risk surface obtained by convolution with a Gaussian kernel: in that case, every point on this surface represents the average risk of a network obtained by perturbing the network parameters according to a Gaussian distribution. The local entropy also relates to a perturbation, but the perturbation is either accepted or rejected based upon its relative performance (as measured by the exponentiated loss) compared with typical perturbations. Thus the local entropy perturbation concentrates on regions of weight space with low empirical risk, provided they have sufficient probability mass under the distribution of the random perturbation. Section 4 yields further insight into the local entropy function. Input: DISPLAYFORM0 w, µ ← w 3:for i ∈ {1, ..., L} do Run SGLD for L iterations.4: DISPLAYFORM1 Entropy-SGLD onlyStep along stochastic local entropy ∇ BID43, which generates an exact sample in the limit of infinite computation and requires that the empirical risk be differentiable. 2 The final output of Entropy-SGD is the deterministic predictor corresponding to the final weights w * achieved by several epochs of optimization. Algorithm 1 gives a complete description of the stochastic gradient step performed by Entropy-SGD. If we rescale the learning rate, η ← 1 2 η τ, lines 6 and 7 are equivalent to DISPLAYFORM2 Notice that the noise term is multiplied by a factor of 2/τ. This follows from the definition of the local entropy. A multiplicative factor ε-called the "thermal noise", but playing exactly the same role as 2/τ here-appears in the original description of the Entropy-SGD algorithm given by Chaudhari et al. However, ε does not appear in the definition of local entropy used in their stability analysis. Our derivations highlights that the scaling the noise term in SGLD update has a profound effect: the thermal noise exponentiates the density that defines the local entropy. The smoothness analysis of Entropy-SGD does not take into consideration the role of ε, which is critical because Chaudhari et al. take ε to be as small as 10 −3 and 10 −4. Indeed, the that the local entropy surface is smoother no longer holds. We will see that τ controls the differential privacy and thus the generalization error of Entropy-SGD. Let Q, P be probability measures defined on R p, assume Q is absolutely continuous with respect to P, and write dQ dP: R p → R + ∪ {∞} for some Radon-Nikodym derivative of Q with respect to P. Then the Kullback-Liebler divergence (or relative entropy) of P from Q is defined to be DISPLAYFORM0 For p, q ∈, we will abuse notation and define DISPLAYFORM1 where B(p) denotes the Bernoulli distribution on {0, 1} with mean p. We now present a PAC-Bayes theorem, first established by BID31. We focus on the setting of bounding the generalization error of a (randomized) classifier on a finite discrete set of labels K. The following variation is due to BID28 for 0-1 loss (see also BID26 and BID9 .) Theorem 3.1 (PAC-Bayes BID31 BID28). Under 0-1 loss, for every δ > 0, m ∈ N, distribution D on R k × K, and distribution P on R p, DISPLAYFORM2 We will also use the following variation of a PAC-Bayes bound, where we consider any bounded loss function. Theorem 3.2 (Linear PAC-Bayes Bound (; BID9). Fix λ > 1/2 and assume the loss takes values in an interval of length L max. For every δ > 0, m ∈ N, distribution D on R k × K, and distribution P on R p, DISPLAYFORM3 We introduce several additional generalization bounds when we introduce differential entropy. We now present our first contribution, a connection between the local entropy and PAC-Bayes bounds. We begin with some notation for Gibbs distributions. For a measure P on R p and function g: R p → R, let P[g] denote the expectation g(h)P(dh) and, provided P[g] < ∞, let P g denote the probability measure on R p, absolutely continuous with respect to P, with Radon-Nikodym derivative DISPLAYFORM0. A distribution of the form P exp(−τg) is generally referred to as a Gibbs distribution. In the special case where P is a probability measure, we call P exp(−τR S) a "Gibbs posterior". for some λ > 1/2, and let P be a multivariate normal distribution with mean w and covariance matrix (τγ) −1 I p. Then maximizing the local entropy F γ,τ (w; S) with respect to w is equivalent to minimizing a linear PAC-Bayes bound (Theorem 3.2) on the risk R D (G w,S γ,τ) of the Gibbs posterior G w,S γ,τ = P exp(−τR S), where the bound is optimized with respect to the mean w of P.Proof. Let m, δ, D, and P be as in Theorem 3.1 and let S ∼ D m. The linear PAC-Bayes bound (Theorem 3.2) ensures that for any fixed λ > 1/2 and bounded loss function, with probability at least 1 − δ over the choice of S, the bound DISPLAYFORM1 holds for all Q ∈ M 1 (R p). Minimizing the upper bound on the risk R D (Q) of the randomized classifier Q is equivalent to the program DISPLAYFORM2 with r(h) = m λ L maxR S (h). By (, Lem. 1.1.3), for all Q ∈ M 1 (R p) with KL(Q||P) < ∞, DISPLAYFORM3 Using Eq., we may reexpress Eq. as DISPLAYFORM4 By the nonnegativity of the Kullback-Liebler divergence, the infimum is achieved when the KL term is zero, i.e., when Q = P exp(−r). Then DISPLAYFORM5 Finally, it is plain to see that F γ,τ (w; DISPLAYFORM6, and P = N (w, (τγ) −1 I p ) is a multivariate normal with mean w and covariance matrix (τγ) −1 I.The analysis falls short when the loss function is unbounded, because the PAC-Bayes bound we have used applies only to bounded loss functions. BID19 described PAC-Bayes generalization bounds for unbounded loss functions. (See BID21 for related work on excess risk bounds and further references). For their bounds to be evaluated on the negative log likelihood loss, one needs some knowledge of the data distribution in order to approximate certain statistics of the deviation of the empirical riskR S (w) from true risk R D (w).5 DATA-DEPENDENT PAC-BAYES PRIORS VIA DIFFERENTIAL PRIVACY Theorem 4.1 reveals that Entropy-SGD is optimizing a PAC-Bayes bound with respect to the prior. As a , the prior P depends on the sample S, and the hypotheses of the PAC-Bayes theorem (Theorem 3.1) are not met. Naively, it would seem that this interpretation of Entropy-SGD cannot explain its ability to generalize. Using tools from differential privacy BID13, we show that if the prior term is optimized in a differentially private way, then a PAC-Bayes theorem still holds, at the cost of a slightly looser bound. We will assume basic familiarity with differential privacy, but give basic definitions and in Appendix A. We use the notation A: Z T for a (randomized) algorithm that takes as input an element in Z and produces an output in T.The key we will employ is due to Dwork et al. (2015b, Thm. 11). Theorem 5.1. Let m ∈ N, let A: Z m T, let D be a distribution over Z, let β ∈, and, for each t ∈ T, fix a set R(t) ⊆ Z m such that P S∼D m (S ∈ R(t)) ≤ β. If A is ε-differentially private for ε ≤ ln(1/β) /(2m), then P S∼D m (S ∈ R(A (S))) ≤ 3 β.Using Theorem 5.1, one can compute tail bounds on the generalization error of fixed classifiers, and then, provided that a classifier is learned from data in a differentially private way, the tail bound holds on the classifier, with less confidence. The following two tail bounds are examples of this idea. The first is a simple variant of (b, Thm. 9) due to Oneto, Ridella, and Anguita (2017, Lem. 2). Theorem 5.2. Let m ∈ N and let A: Z m R p be ε-differentially private. , Lem. 3)). Let m ∈ N and let A: Z m R p be ε-differentially private. Then DISPLAYFORM7 DISPLAYFORM8 The PAC-Bayes theorem allows one to choose the prior based on the data-generating distribution D, but not on the data S ∼ D m. Using differential privacy, we can consider a data-dependent prior P(S).Theorem 5.4. Under 0-1 loss, for every δ > 0, m ∈ N, distribution D on R k ×K, and ε-differentially private data-dependent prior P: DISPLAYFORM0 It follows from the PAC-Bayes theorem (Theorem 3.1) that P S∼D m (S ∈ R(P)) ≤ β. Theorem 5.1 implies that the bound holds with P replaced by P(S), provided that we inflate the probability of failure. In particular, let δ = 3 β. Then ln(1/β) = 2 ln(3/δ). By Theorem 5.1, provided 2mε 2 ≤ ln(1/β), then P S∼D m (S ∈ R(P(S))) ≤ δ. It follows that, with probability no more than δ over S ∼ D m, there exists a distribution Q on R p such that DISPLAYFORM1 The bound stated in Eq. follows immediately. Note that the bound holds for any posterior Q, including one obtained by optimizing a different PAC-Bayes bound. We have chosen to present a differentially private version of Theorem 3.1 rather than Theorem 3.2, because the former tends to be tighter numerically. Giving a differentially private version of Theorem 3.2, or any other PAC-Bayes bound, should be straightforward: one merely needs to decide how to incorporate the constraint between ε, β, and m in Theorem 5.1. We have chosen to deal with the constraint via a max operation affecting the width of the confidence interval. Note that, in realistic scenarios, δ is large enough relative to ε that an ε-differentially private prior P(S) contributes 2ε 2 to the generalization error. Therefore, ε must be much less than one to not contribute a nontrivial amount to the generalization error. In order to match the m −1 rate by which the KL term decays, one must have ε ∈ O(m −1/2). Our empirical studies use this rate. We have already explained that the weights learned by Entropy-SGD can be viewed as the mean of a data-dependent prior P(S). By Theorem 5.4 and the fact that post-processing does not decrease privacy, it would suffice to establish that the mean is ε-differentially private in order to obtain a risk bound on the corresponding Gibbs posterior classifier. Entropy-SGD can be viewed as stochastic gradient ascent on the negative local entropy, but with biased gradient estimates. The bias comes from the use of SGLD to compute the expectation in Eq.. Putting aside this issue, existing privacy analyses of SGD worsen after every iteration. For the number of iterations necessary to obtain reasonable weights, known upper bounds on the differential privacy of SGD yield vacuous generalization bounds. The standard (if idealized) approach for optimizing a data-dependent objective in a private way is to use the exponential mechanism BID33. In the context of maximizing the local entropy, the exponential mechanism correspond to sampling exactly from the "local entropy (Gibbs) distribution" DISPLAYFORM0 where β > 0 and P is some measure on R p. (It is natural to take P to be Lebesgue measure, or a multivariate normal distribution, which would correspond to L2 regularization of the local entropy.)The following establishes the privacy of a sample from the local entropy distribution: Theorem 5.5. Let γ, τ > 0, and assume the range of the loss is contained in an interval of length L max. One sample from the local entropy distribution P exp(β F γ,τ (·;S)), is 2β L max τ m -differentially private. Proof. The follows immediately from the following two lemmas. Lemma 5.6 ((, Thm. 6)). Let q: Z m × R p → R be measurable, let P be a measure on R p, let β > 0, and assume P[exp(−β q(S, ·))] < ∞ for all S ∈ Z m. Let ∆q def = sup S,S sup w∈R p |q(S, w) − q(S, w)|, where the first supremum ranges over pairs S, S ∈ Z m that disagree on no more than one coordinate. Let A: Z m R p, on input S ∈ Z m, output a sample from the Gibbs distribution P exp(−β q(S,·)). Then A is 2β ∆q-differentially private. Lemma 5.7. Let F γ,τ (w; S) be defined as Eq., assume the range of the loss is contained in an interval of length L max, and define q(S, w) = −F γ,τ (w; S). Then ∆q DISPLAYFORM1 Proof. The proof essentially mirrors that of (, Thm. 6).There are two obvious obstructions to using the exponential mechanism to pick a prior mean: first, cross-entropy loss can change in an unbounded way when swapping a single data point; second, sampling from the local entropy distribution exactly is hard in general. To sidestep the first obstruction, we modify the underlying cross-entropy loss to be bounded by rescaling the probabilities output by the classifier to be bounded away from zero and one, allowing us to invoke Lemma 5.7. (Details of our modification of the cross entropy are described in Appendix B.2.1.)There is no simple way to sidestep the second obstruction. Instead, we once again use SGLD to generate an approximate sample from the local entropy distribution. In summary, to optimize the local entropy F γ,τ (·; S) in a private way to obtain the prior mean w, we repeatedly perform the SGLD update DISPLAYFORM2 where at each roundĝ(w) is an estimate of the gradient ∇ w F γ,τ (w; S). (Recall the identity Eq..) As in Entropy-SGD, we construct biased gradient estimates via an inner loop of SGLD. In summary, the only change to Entropy-SGD is the addition of noise in the outer loop. We call the ing algorithm Entropy-SGLD. (See Algorithm 1. Note that we take β = 1 in our experiments.)There have been a number of privacy analyses of SGLD BID35 BID6 BID12 BID42 BID34. Most of these analyses deliver (ε, δ)-differential privacy, but none of them take advantage of the fact that SGLD mixes in the limit as it converges weakly to the Gibbs distributions, under certain technical conditions (, Thm. 7). In our analysis and bound calculations, we therefore make the approximation that SGLD has the same privacy as its limiting invariant measure, the exponential mechanism. Building a less conservative model of the privacy of SGLD is an open problem. However, by making this approximation, we may see the potential/limits of combining differentially private optimization and PAC-Bayesian bounds. We return to the issues again in light of our empirical findings (Section 6) and in our discussion (Section 7). The generalization bounds that we have devised are data-dependent and so the question of their utility is an empirical one that requires data. In this section, we perform an empirical study of SGD, SGLD, Entropy-SGD, and Entropy-SGLD on the MNIST data set, on both convolutional and fully connected architectures, and compare our generalization bounds to estimates based on held-out data. Under our privacy approximation, SGLD and Entropy-SGLD are ε-differentially private and we take advantage of this fact to apply differentially private versions of two tail bounds and our PACBayes bound. The degree ε of privacy is determined by the τ parameter of the local entropy (C.f. thermal noise 2/τ), and then, in turn, ε contributes to our bounds on the generalization error. As theory predicts, τ affects the degree of overfitting empirically, and no bound we compute is violated too frequently. Of course, the validity of our generalization bounds rests on the degree to which our privacy approximation is violated. 3 We reflect on our approximation in light of our empirical , and then return to this point in the discussion. The weights learned by SGD, SGLD, and Entropy-SGD are treated differently from those learned by Entropy-SGLD. In the former case, the weights parametrize a neural network as usual, and the training and test error are computed using these weights. In the latter case, the weights are taken to be the mean of a multivariate normal prior, and we evaluate the training and test error of the The gap is an estimate of the generalization error. On true labels, SGLD finds classifiers with relatively small generalization error. At low thermal noise settings, SGLD (and its zero limit, SGD), achieve small empirical risk. As we increase the thermal noise, the empirical 0-1 error increases, but the generalization error decreases. At 0.1 thermal noise, risk is close to 50%. (top-right) On random labels, SGLD has high generalization error for thermal noise values 0.01 and below. (True error is 50%). (middle-left) On true labels, Entropy-SGD, like SGD and SGLD, has small generalization error. For the same settings of thermal noise, empirical risk is lower. (middle-right) On random labels, Entropy-SGD overfits for thermal noise values 0.005 and below. Thermal noise 0.01 produces good performance on both true and random labels. (bottom row) Entropy-SGLD is configured to be ε-differentially private with ε ≈ 0.0327 by setting τ = √ m, where m is the number of training samples. (bottom-left) On true labels, the generalization error for networks learned by Entropy-SGLD is close to zero. Generalization bounds are relatively tight. (bottom-right) On random label, Entropy-SGLD does not overfit. See Fig. 3 for SGLD bounds at same privacy setting. associated Gibbs posterior (i.e., a randomized classifier). We also report the performance of the (deterministic) network parametrized by these weights (called the "mean" classifier) in order to give a coarse statistic summarizing the local empirical risk surface. Following BID44, we study these algorithms on MNIST with its original ("true") labels, as well as on random labels. Parameter τ that performs very well in one setting often does not perform well in the other. Random labels mimic data where the Bayes error rate is high, and where overfitting can have severe consequences. We use a two-class variant of MNIST . 4 (See FIG7 and Appendix C for our experiments on the standard multiclass MNIST dataset. They yield similar insight.) Some experiments involve random labels, i.e., labels drawn independently and uniformly at random at the start of training. We study three network architectures, abbreviated FC600, FC1200, and CONV. Both FC600 and FC1200 are 3-layer fully connected networks, with 600 and 1200 units per hidden layer, respectively. CONV is a convolutional architecture. All three network architectures are taken from the MNIST experiments by BID10, but adapted to our two-class version of MNIST. 5 Let S and S tst denote the training and test sets, respectively. For all learning algorithms we track (i)R S (w) andR S tst (w), i.e., the training and test error for w. We also track DISPLAYFORM0.e., the mean training and test error of the local Gibbs distribution, viewed as a randomized classifier ("Gibbs") and, using the differential privacy bounds in Theorem 5.5, compute (iii) a PAC-Bayes bound on R D (G w,S γ,τ) using Theorem 5.4 ("PAC-bound"); (iv) the mean of a Hoeffding-style bound on R D (w), where w ∼ P exp(F γ,τ (·;S)),, using Theorem 5.2 ("H-bound");(v) an upper bound on the mean of a Chernoff-style bound on R D (w), where w ∼ P exp(F γ,τ (·;S)),, using Theorem 5.3 ("C-bound").We also compute H-and C-bounds for SGLD, viewed as a sampler for w ∼ P exp(−τR S), where P here is Lebesgue measure. In order for SGLD and Entropy-SGLD to be private, we modify the cross entropy loss function to be bounded. We achieve this by an affine transformation of the neural networks output that prevents extreme probability (se Appendix B.2.1). With the choice of τ = √ m, and the loss function taking values in an interval of length L max = 4, Entropy-SGLD is ε-differentially private, with ε ≈ 0.0327. See Appendix B.2 for additional details. Note that, in the calculation of (iii), we do not account for Monte Carlo error in our estimate ofR S (w). The effect is small, given the large number of iterations of SGLD performed for each point in the plot. Recall that DISPLAYFORM1 and so we may interpret the bounds in terms of the performance of a randomized classifier or the mean performance of a randomly chosen classifier. Key for the convolutional architecture (CONV) appear in FIG2. Results for FC600 and FC1200 appear in Fig. 2 of Appendix B. (Training the CONV network produces the lowest training/test errors and tightest generalization bounds. Results and bounds for FC600 are nearly identical to those for FC1200, despite FC1200 having three times as many parameters.)The top row of FIG2 presents the performance of SGLD for various levels of thermal noise 2/τ under both true and random labels. (Under our privacy approximation, we may also use SGLD to directly perform a private optimization of the empirical risk surface. The level of thermal noise determines the differential privacy of SGLD and so we expect to see a tradeoff between empirical risk and generalization error. Note that SGD is the same as SGLD with zero thermal noise.) SGD achieves the smallest training and test error on true labels, but overfits the worst on random labels. In comparison, SGLD's generalization performance improves with higher thermal noise, while its risk performance worsens. At 0.05 thermal noise, SGLD achieves reasonable but relatively large risk but almost zero generalization error on both true and random labels. Other thermal noise settings have either much worse risk or generalization performance. The middle row of FIG2 presents the performance of Entropy-SGD for various levels of thermal noise 2/τ under both true and random labels. As with SGD, Entropy-SGD's generalization performance improves with higher thermal noise, while its risk performance worsens. At the same levels of thermal noise, Entropy-SGD outperforms the risk and generalization error of SGD. At 0.01 thermal noise, Entropy-SGD achieves good risk and low generalization error on both true and random labels. However, the test-set performance of Entropy-SGD at 0.01 thermal noise is still worse than that of SGD. Whether this difference is due to SGD overfitting to the MNIST test set is unclear and deserves further study. The bottom row of FIG2 presents the performance of Entropy-SGLD with τ = √ m on true and random labels. (This corresponds to approximately 0.09 thermal noise.) On true lables, both the mean and Gibbs classifier learned by Entropy-SGLD have approximately 2% test error and essentially zero generalization error, which is less than predicted by our bounds. Our PAC-Bayes risk bounds are roughly 3%. As expected by the theory, Entropy-SGLD does not overfit on random labels, even after thousands of epochs. We find that our PAC-Bayes bounds are generally tighter than the H-and C-bounds. All bounds are nonvacuous, though still loose. The error bounds reported here are tighter than those reported by BID18. However, the validity of all three privacy-based bounds that we report rests on the privacy approximation regarding SGLD, and so interpreting these bounds requires some subtlety. We achieve much tighter generalization bounds than previously reported, and better test error, but we are still far from the performance of SGD. This is despite making a strong approximation, and so we might view these as telling us the limits of combining differential privacy and PAC-Bayes bounds. Weaker notions of stability/privacy may be necessary to achieve further improvement in generalization error and test error. Despite the coarse privacy approximation, no bound is ever violated: possible explanations include the bounds simply being loose and/or the data being far from worst case. Note that, given the number of experiments, we might even expect a violation for tight bounds. Indeed, our performance on random labels supports the hypothesis that the privacy of (Entropy-)SGLD does not degrade over time, at least not in a way that can be detected by our experiments. Our work reveals that Entropy-SGD can be understood as optimizing a PAC-Bayes generalization bound in terms of the bound's prior. Because the prior must be independent of the data, the bound is invalid, and, indeed, we observe overfitting in our experiments with Entropy-SGD when the thermal noise 2/τ is set to 0.0001 as suggested by Chaudhari et al. for MNIST. PAC-Bayes priors can, however, depend on the data distribution. This flexibility seems wasted, since the data sample is typically viewed as one's only view onto the data distribution. However, using differential privacy, we can span this gap. By performing a private computation on the data, we can extract information about the underlying distribution, without undermining the statistical validity of a subsequent PAC-Bayes bound. Our PAC-Bayes bound based on a differentially private prior is made looser by the use of a private data-dependent prior, but the gains in choosing a datadistribution-dependent prior more than make up for the expansion of the bound due to the privacy. (The gains come from the KL term being much smaller on the account of the prior being better matched to the posterior.) Understanding how our approach compares to local PAC-Bayes priors BID9 is an important open problem. The most elegant way to make Entropy-SGD private is to replace SGD with a sample from the Gibbs distribution (known as the exponential mechanism in the differential privacy literature). However, generating an exact sample is intractable, and so practicioners use SGLD to generate an approximate sample, relying on the fact that SGLD converges weakly to the exponential mechanism under certain technical conditions. Our privacy approximation allows us to proceed with a theoretical analysis by assuming that SGLD achieves the same privacy as the exponential mechanism. On the one hand, we do not find overt evidence that our approximation is grossly violated. On the other, we likely do not require such strong privacy in order to control generalization error. We might view our privacy-based bounds as being optimistic and representing the bounds we might be able to achieve rigorously should there be a major advance in private optimization. (No analysis of the privacy of SGLD takes advantage of the fact that it mixes weakly.) On the account of using private data-dependent priors, our bounds are significantly tighter than those reported by BID18. However, despite our bounds potentially being optimistic, the test set error we are able to achieve is still 5-10 times that of SGD. Differential privacy may be too conservative for our purposes, leading us to underfit. Indeed, we think it is unlikely that Entropy-SGD has strong differential privacy, yet we are able to achieve good generalization on both true and random labels under 0.01 thermal noise. Identifying the appropriate notion of privacy/stability to combine with PAC-Bayes bounds is an important problem. Despite our progress on building learning algorithms with strong generalization performance, and identifying a path to much tighter PAC-Bayes bounds, Entropy-SGLD learns much more slowly than Entropy-SGD, the risk of Entropy-SGLD is far from state of the art, and our PAC-Bayes bounds are loose. It seems likely that there is a fundamental tradeoff between the speed of learning, the excess risk, and the ability to produce a certificate of one's generalization error via a rigorous bound. Characterizing the relationship between these quantities is an important open problem. A : DIFFERENTIAL PRIVACY Here we formally define some of the differential privacy related terms used in the main text. (See BID13 BID15 for more details.) Let U,U 1,U 2,... be independent uniform random variables, independent also of any random variables introduced by P and E, and let π: DISPLAYFORM0 Definition A.1. A randomized algorithm A from R to T, denoted A: R T, is a measurable map A: × R → T. Associated to A is a (measurable) collection of random variables {A r : r ∈ R} that satisfy A r = A (U, r). When there is no risk of confusion, we will write A (r) for A r. Definition A.2. A randomized algorithm A: Z m T is (ε, δ)-differentially private if, for all pairs S, S ∈ Z m that differ at only one coordinate, and all measurable subsets B ⊆ T, we have P(A (S) ∈ B) ≤ e ε P(A (S) ∈ B) + δ.We will write ε-differentially private to mean (ε, 0)-differentially private algorithm. Definition A.3. Let A: R T and A: DISPLAYFORM1 Lemma A.4 (post-processing). Let A: Z m T be (ε, δ)-differentially private and let F: T T be arbitrary. Then F • A is (ε, δ)-differentially private. We studied three architectures: CONV, FC600, and FC1200.CONV was a convolutional neural network, whose architecture was the same as that used by BID10 for multiclass MNIST classification, except modified to produce a single probability output for our two-class variant of MNIST. In particular, CONV has two convolutional layers, a fully connected ReLU layer, and a sigmoidal output layer, yielding 126, 711 parameters in total. FC600 and FC1200 are fully connected 3-layer neural networks, with 600 and 1200 hidden units, respectively, yielding 834, 601 and 2, 385, 185 parameters in total, respectively. We used ReLU activations for all but the last layer, which was sigmoidal to produce an output in.In their MNIST experiments, BID10 use dropout and batch normalization. We did not use dropout. The bounds we achieved with and without batch norm were very similar. Without batch norm, however, it was necessary to tune the learning rates. Understanding the combination of SGLD and batch norm and the limiting invariant distribution, if any, is an important open problem. B.2.1 OBJECTIVE All networks are trained to minimize a bounded variant of empirical cross entropy loss. The change involves replacing g(p, y) = − log p with g(p, y) = − log ψ(p), where DISPLAYFORM0 is an affine transformation that maps to DISPLAYFORM1, removing extreme probability values. As a , the binary cross entropy loss BCE is contained in an interval of length L max. In particular, DISPLAYFORM2 We take L max = 4 in our experiments. Ordinarily, an epoch implies one pass through the entire data set. For SGD, each stochastic gradient step processes a minibatch of size K = 128. Therefore, an epoch is m/K = 468 steps of SGD. An epoch for Entropy-SGD and Entropy-SGLD is defined as follows: each iteration of the inner SGLD Test (On true labels, SGLD learns a network with approximately 3% higher training and test error than the mean and Gibbs networks learned by Entropy-SGLD. SGLD does not overfit on random labels, as predicted by theory. The C-bound on the true error of this network is around 8%, which is worse than the roughly 4% C-bound on the mean classifier.loop processes a minibatch of size K = 128, and the inner loop runs for L = 20 steps. Therefore, an epoch is m/(LK) steps of the outer loop. In concrete terms, there are 20 steps of SGD per every one step of Entropy-SG(L)D. Concretely, the x-axis of our plots measure epochs divided by L. This choice, used also by BID10, ensures that the wall-clock time of Entropy-SG(L)D and SGD align. The step sizes for SGLD must be square summable but not summable. The step sizes for the outer SGLD loop are of the form η t = ηt −0.6, with η = 0.006 γτ. The step sizes for the inner SGLD loop are of the form η t = ηt −1, with η = 1 γτ. The estimate produced by the inner SGLD loop is computed using a weighted average (line 8) with α = 0.75. We use SGLD again when computing the PAC-Bayes generalization bound (Appendix B.3.2). In this case, SGLD is used to sample from the local Gibbs distribution when estimating the Gibbs risk and the KL term. We run SGLD for 1000 epochs to obtain our estimate. Again, we use weighted averages, but with α = 0.005, in order to average over a larger number of samples and better control the variance. We set γ = 1 and τ = √ m and keep the values fixed during optimization. By Theorem 5.5, the value of τ, L max, and β determine the differential privacy of Entropy-SGLD. In turn, the differential privacy parameter ε and confidence parameter δ contribute When the empirical error is close to zero, the KL version of the PAC-Bayes bound Theorem 3.1 is considerably tighter than the Hoeffding-style bound first described by BID31. However, using this relative entropy bound requires one to be able to compute the largest value p such that KL(q||p) ≤ c. There does not appear to be a simple formula for this value. In practice, however, the value can be efficiently numerically approximated using, e.g., Newton's method. See (, §2.2 and App. B). Let (w) = τR S (w). By (, Lem Both terms have obvious Monte Carlo estimates: DISPLAYFORM0 where w 1, . . ., w k are taken from a Markov chain targeting P exp(−), such as SGLD run for k 1 steps (which is how we computed our bounds), and log P[exp(−)] = log exp{− (w)} P(dw) DISPLAYFORM1 where h 1,..., h k are i.i.d. P (which is a multivariate Gaussian in this case). In the latter case, due to the concavity of log, the estimate is a lower bound with high probability, yielding a high probability upper bound on the KL term. We evaluate the same generalization bounds on the standard MNIST classification task as in the MNIST binary labelling case. All the details of the network architectures and parameters are as stated in Appendix B.2, with two exception: following BID10, we use a fully connected network with 1024 hidden units per layer, denoted FC1024. The neural network produces a probability vector (p 1, . . ., p K) via a soft-max operation. Ordinarily, we then apply the cross entropy loss corresponding to g FIG2.., p K ), y) = − log p y. When training privately, we use a bounded variant of the cross entropy loss, where the function g above is replaced by g((p 1, . . ., p K), y) = − log ψ(p y), and ψ is defined as in Eq.. FC1024 network trained on true labels. The train and test error suggest that the generalization gap is close to zero, while all three bounds exceed the test error by slightly more than 3%. (bottomleft) CONV network trained on true labels. Both the train and the test errors are lower than those achieved by the FC1024 network. We still do not observe overfitting. The C-bound and PAC-Bayes bounds exceed the test error by ≈ 3%. (top-right) FC1024 network trained on random labels. After approximately 1000 epochs, we notice overfitting by ≈ 2%. Running Entropy-SGLD further does not cause an additional overfitting. Theory suggests that our choice of τ prevents overfitting via differential privacy. (bottom-right) CONV network trained on random labels. We observe almost no overfitting (less than 1%). Both training and test error coincide and remain close to the guessing rate (90%).

We show that Entropy-SGD optimizes the prior of a PAC-Bayes bound, violating the requirement that the prior be independent of data; we use differential privacy to resolve this and improve generalization.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the following document.

Question: In this paper, we investigate learning the deep neural networks for automated optical inspection in industrial manufacturing. Our preliminary has shown the stunning performance improvement by transfer learning from the completely dissimilar source domain: ImageNet. Further study for demystifying this improvement shows that the transfer learning produces a highly compressible network, which was not the case for the network learned from scratch. The experimental shows that there is a negligible accuracy drop in the network learned by transfer learning until it is compressed to 1/128 reduction of the number of convolution ﬁlters. This is contrary to the compression without transfer learning which loses more than 5% accuracy at the same compression rate. in, the network trained from scratch can also achieve 99.78% accuracy with the extensively 23 augmented data and a long period of training. Surprisingly, however, although the performance of 24 both networks is similar, the network trained from scratch learns much denser features of the input 25 data than the network trained by transfer learning. We experimentally show that using standard 26 teacher-student model compression technique, the network trained by transfer learning can be 27 compressed to 1/128 reduction of the number of convolution filters with a negligible accuracy drop. In contrast, the network trained from scratch loses more than 5% accuracy at the same compression 29 rate. Previous works on the network compression have focused on the methods of compression, but our work is the first report to the best of our knowledge that the transfer learning is 31 related to the network compression. The rest of the paper is organized as follows. In Section 2, we explain the method of experiments and show the compression in Section 2.3. Finally, we conclude with a brief discussion in Section 3. a given input image, we can see that the TL teacher network learns much sparser features of the 64 input data than the Scratch teacher network (Figure 2). The sparsity of activation in the TL teacher

Represent the TLDR for finding the corresponding scientific article

We experimentally show that transfer learning makes sparse features in the network and thereby produces a more compressible network.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.