Question: Represent the text to find a very short summary (1-2 sentences max):

Question: Federated learning involves training and effectively combining machine learning models from distributed partitions of data (i.e., tasks) on edge devices, and be naturally viewed as a multi- task learning problem. While Federated Averaging (FedAvg) is the leading optimization method for training non-convex models in this setting, its behavior is not well understood in realistic federated settings when the devices/tasks are statistically heterogeneous, i.e., where each device collects data in a non-identical fashion. In this work, we introduce a framework, called FedProx, to tackle statistical heterogeneity. FedProx encompasses FedAvg as a special case. We provide convergence guarantees for FedProx through a device dissimilarity assumption. Our empirical evaluation validates our theoretical analysis and demonstrates the improved robustness and stability of FedProx for learning in heterogeneous networks. Large networks of remote devices, such as phones, vehicles, and wearable sensors, generate a wealth of data each day. Federated learning has emerged as an attractive paradigm to push the training of models in such networks to the edge . In such settings, the goal is to jointly learn over distributed partitions of data/tasks, where statistical heterogeneity and systems constraints present significant challenges. Optimization methods that allow for local updating and low participation have become the de facto solvers for federated learning . These methods perform a variable number of local updates on a subset of devices to enable flexible and efficient communication. Of current federated optimization methods, FedAvg has become state-of-the-art for non-convex federated learning. However, FedAvg was not designed to tackle the statistical heterogeneity which is inherent in federated settings; namely, that data may be non-identically distributed across devices. In realistic statistically heterogeneous settings, FedAvg has been shown to diverge empirically (, Sec 3), and it also lacks theoretical convergence guarantees. Indeed, recent works exploring convergence guarantees are limited to unrealistic scenarios, where the data is either shared across devices or distributed in an IID (identically and independently distributed) manner, or all devices are active at each communication round (; ; ; ; ;).Due to the statistical heterogeneity of the data in federated networks, one can think of federated learning as a prime example of distributed multi-task learning, where each device corresponds to a task. However, the more common goal of federated learning-and the focus of this work-involves training a single global model on distributed data collected for these various tasks. We introduce and study a novel optimization framework in the federated setting. Our focus on its convergence behavior in the face of statistically heterogeneous data is closely related to the classical multi-task setting which involves jointly learning task-specific models from statistically heterogeneous data. Contributions. We propose a federated optimization framework for heterogeneous networks, FedProx, which encompasses FedAvg. In order to characterize the convergence behavior of FedProx, we invoke a device dissimilarity assumption in the network. Under this assumption, we provide the first convergence guarantees for FedProx. Finally, we demonstrate that our theoretical assumptions reflect empirical performance, and that FedProx can improve the robustness and stability of convergence over FedAvg when data is heterogeneous across devices. Large-scale distributed machine learning has motivated the development of numerous distributed optimization meth-ods in the past decade (see, e.g., BID5 ; a; ; ; ; Richtárik & Takáč, 2016; BID3 . However, it is increasingly attractive to learn statistical models directly over networks of distributed devices. This problem, known as federated learning, requires tackling novel challenges with privacy, heterogeneous data, and massively distributed networks. Recent optimization methods have been proposed that are tailored to the specific challenges in the federated setting. These methods have shown significant improvements over traditional distributed approaches like ADMM BID2 by allowing both for inexact local updating in order to balance communication vs. computation in large networks, and for a small subset of devices to be active at any communication round (; ;). For example, proposes a communication-efficient primal-dual optimization method that learns separate but related models for each device through a multi-task learning framework. However, such an approach does not generalize to non-convex problems, e.g. deep learning, due to lack of strong duality. In the non-convex setting, Federated Averaging (FedAvg), a heuristic method based on averaging local Stochastic Gradient Descent (SGD) updates, has instead been shown to work well empirically .Unfortunately, FedAvg is quite challenging to analyze due to its local updating scheme, the fact that few devices are active at each round, and the issue that data is heterogeneous. Recent works have made steps towards analyzing FedAvg in simpler settings. For instance, parallel SGD and related variants (; ; ; ;), which make local updates similar to FedAvg, have been studied in the IID setting. Although some works (; ;) have recently explored convergence guarantees in heterogeneous settings, they make the limiting assumptions such as full participation of all devices, convexity , or uniformly bounded gradients . There are also several heuristic approaches that aim to tackle statistical heterogeneity, either by sharing the local device data or some server-side proxy data (; ;), which may be unrealistic in practical federated settings. In this section, we introduce the key ingredients behind recent methods for federated learning, including FedAvg, and then outline our proposed framework, FedProx. Federated learning methods (e.g., ; ;) are designed to handle multiple devices collecting data and a central server coordinating the global learning objective across the network. The aim is to minimize: DISPLAYFORM0 where N is the number of devices, p k ≥ 0, ∀k, and k p k =1. In general, the local objectives measure the local empirical risk over possibly differing data distributions DISPLAYFORM1, with n k samples available at each device k. Hence, we can set p k = n k n, where n= k n k is the total number of data points. To reduce communication and handle systems constraints, federated optimization methods commonly allow for low participation and local updating. At each round, a subset of the devices are selected and use local solvers to optimize the local objectives. Then the local updates are aggregated via a central server. Each of the local objectives can be solved inexactly, as formally defined below. Definition 1 (γ-inexact solution). For a function h(w; w 0) = F (w) + µ 2 w − w 0 2, and γ ∈, we say w * is a γ-inexact solution of min w h(w; w 0), if ∇h(w * ; w 0) ≤ γ ∇h(w 0 ; w 0), where ∇h(w; w 0) = ∇F (w) + µ(w − w 0). Note that a smaller γ corresponds to higher accuracy. We use γ-inexactness in our analysis (Section 4) to measure the amount of local computation from each local solver. In experiments (Section 5), we simply run an iterative local solver for some number of local epochs, which can be seen as a proxy for γ-inexactness. In Federated Averaging (FedAvg) , at each round, a subset K N of devices are selected and run SGD locally for E number of epochs to optimize the local objective F k on device k, and then the ing model updates are averaged. shows empirically that it is crucial to tune the number of local epochs for FedAvg to converge, as additional local epochs allow local models to move further away from the initial global model, potentially causing divergence. Thus, it is beneficial to restrict the amount of local deviation through a more principled tool than heuristically limiting the number of local epochs of some iterative solver. This serves as our inspiration for FedProx, introduced below. Instead of just minimizing the local function F k, in FedProx, device k uses its local solver to approximately minimize the following surrogate objective h k: DISPLAYFORM0 The proximal term in the above expression effectively limits the impact of local updates by restricting them to be close to the current model w t. We note that proximal terms such as the one above are a popular tool utilized throughout the optimization literature (see Appendix C). An important distinction of the proposed usage is that we suggest, explore, and analyze such a term for the purpose of tackling statistical heterogeneity in federated settings. DISPLAYFORM1 Server selects a subset S t of K devices at random (each device k is chosen with probability p k); Server sends w t to all chosen devices; Each chosen device k ∈ S t finds a w t+1 k which is a γ-inexact minimizer of: DISPLAYFORM2 Each chosen device k sends w t+1 k back to the server; Server aggregates the w's as w DISPLAYFORM3 In Section 4, we see that the usage of the proximal term makes FedProx more amenable to theoretical analysis. In Section 5, we also see the modified local subproblem in FedProx in more robust and stable convergence compared to FedAvg for heterogeneous datasets. Note that FedAvg is a special case of FedProx with µ = 0. In this section we first introduce a metric that specifically measures the dissimilarity among local functions. We call this metric local dissimilarity. We then analyze FedProx under an assumption on bounded local dissimilarity. Definition 2 (B-local dissimilarity). The local functions DISPLAYFORM0 for ∇f (w) = 0.Here E k [·] denotes the expectation over devices with masses p k =n k /n and N k=1 p k =1. Note that B(w)≥ 1 and the larger the value of B(w), the larger is the dissimilarity among the local functions. Moreover, if F k (·)'s are associated with empirical risk objectives and the samples on all the devices are homogeneous, then B(w) → 1 for every w as all the local functions converge to the same expected risk function. Interestingly, similar assumptions (e.g., ;) have been explored elsewhere for differing purposes; see more in Appendix C. Using Definition 2, we now state our formal dissimilarity assumption, which we use in our convergence analysis. Assumption 1 (Bounded dissimilarity). For some > 0, there exists a B such that for all the points w ∈ S c = {w | ∇f (w) 2 > }, B(w) ≤ B.Using Assumption 1, we analyze the amount of expected objective decrease if one step of FedProx is performed. Theorem 3 (Non-convex FedProx Convergence: B-local dissimilarity). Let Assumption 1 hold. Assume the functions F k are non-convex, L-Lipschitz smooth, and there exists DISPLAYFORM1 t is not a stationary solution and the local functions F k are B-dissimilar, i.e. B(w t) ≤ B. If µ, K, and γ in Algorithm 1 are chosen such that DISPLAYFORM2 then at iteration t of Algorithm 1, we have the following expected decrease in the global objective: DISPLAYFORM3 where S t is the set of K devices chosen at iteration t. We direct the reader to Appendix A.1 for a detailed proof. Theorem 3 uses the dissimilarity in Definition 2 to identify sufficient decrease at each iteration for FedProx. In Appendix A.2, we provide a corollary characterizing the performance with a more common (though slightly more restrictive) bounded variance assumption. Remark 4. In order for ρ in Theorem 3 to be positive, we need γB < 1. Moreover, we also need DISPLAYFORM4 These conditions help to quantify the trade-off between dissimilarity bound (B) and the algorithm parameters (γ, K).Finally, we can use the above sufficient decrease to characterize the rate of convergence under Assumption 1. Note that these hold for general non-convex F k (·).Theorem 5 (Convergence rate: FedProx). Given some > 0, assume that for B ≥ B, µ, γ and K the assumptions of Theorem 3 hold at each iteration of FedProx. DISPLAYFORM5 While the thus far hold for non-convex F k (·), we prove the convergence for convex loss in Appendix A.3. To help provide context for the rate in Theorem 5, we compare it with SGD in the convex case in Appendix A.4, Remark 9. We now present empirical for FedProx. We study the effect of statistical heterogeneity on the convergence of FedAvg and FedProx, explore properties of the FedProx framework, and show how empirical convergence relates to the bounded dissimilarity assumption. We show a subset of our experiments here due to space constraints; for full we defer the reader to Appendix B. All code, data, and experiments are publicly available at github.com/litian96/FedProx. Experimental Details. We evaluate FedProx on diverse tasks, models, and both synthetic and real-world datasets. The real datasets are curated from prior work in federated learning (; BID3 . In particular, We study convex models on partitioned MNIST , Federated Extended MNIST BID4 BID3 ) (FEM-NIST), and FMNIST*, and non-convex models on Sentiment140 BID7 ) FORMULA0 Effect of Statistical Heterogeneity. In Figure 1, we study how statistical heterogeneity affects convergence using four synthetic datasets. From left to right, as data become more heterogeneous, convergence becomes worse for FedProx with µ=0 (FedAvg). Setting µ > 0 is particularly useful in heterogeneous settings although that may slow convergence for IID data. Properties of FedProx Framework. The key parameters of FedProx that affect performance are the number of local epochs, E, and the proximal term scaled by µ. We study FedProx under different values of E and µ using the federated datasets described in TAB3 in Appendix B.1. We report the on Shakespeare dataset here and provide similar on all datasets in Appendix B.3. Dependence on E. We explore the effect of E in Figure 2 (left) and show the convergence in terms of the training loss. We see that large E leads to divergence on Shakespeare. In Appendix B.3, we further show that large E leads to similar instability on other heterogeneous datasets. We note here that a large E may be particularly useful in practice when communication is expensive (which is common in federated networks) where small E is prohibitive. In Figure 3, e.g., we show that FedProx with a large E (E=50) and an appropriate µ (µ=0.2) leads to faster and more stable convergence compared with E=1, µ=0 (slow convergence) and E=50, µ=0 (unstable convergence). Figure 1. Effect of data heterogeneity on convergence. We show training loss (see testing accuracy and dissimilarity metric in Appendix B.3, FIG4) on four synthetic datasets whose heterogeneity increases from left to right. The method with µ = 0 corresponds to FedAvg. Increasing heterogeneity leads to worse convergence, but setting µ > 0 can help to combat this. Dependence on µ. We consider the effect of µ on convergence in Figure 2 (middle). We observe that the appropriate µ can force divergent methods to converge or increase the stability for unstable methods (Figure 5, Appendix B.3), thus making the performance of FedProx less dependent on E. In practice, µ can be adaptively chosen based on the current performance of the models. For example, one simple heuristic is to increase µ when seeing the loss increasing and decreasing µ when seeing the loss decreasing. We provide additional experiments demonstrating the effectiveness of this approach in Appendix B.5.Dissimilarity Measurement and Divergence. Finally, in Figure 2 (right), we track the variance of gradients on each device, DISPLAYFORM0, which is lower bounded by B (see Bounded Variance Equivalence Corollary 6). We observe that the dissimilarity metric in Definition 2 is consistent with the training loss. Therefore, smaller dissimilarity indicates better convergence, which can be enforced by setting µ appropriately. Proof. Using our notion of γ-inexactness for each local solver (Definition 1), we can define e t+1 k such that: DISPLAYFORM1 Now let us definew DISPLAYFORM2. Based on this definition, we know DISPLAYFORM3 Let us defineμ = µ − L − > 0 andŵ t+1 k = arg min w h k (w; w t). Then, due to theμ-strong convexity of h k, we have DISPLAYFORM4 Note that once again, due to theμ-strong convexity of h k, we know that DISPLAYFORM5. Now we can use the triangle inequality to get DISPLAYFORM6 Therefore, DISPLAYFORM7 where the last inequality is due to the bounded dissimilarity assumption. Now let us define M t+1 such thatw DISPLAYFORM8 where the last inequality is also due to bounded dissimilarity assumption. Based on the L-Lipschitz smoothness of f and Taylor expansion, we have DISPLAYFORM9 From the above inequality it follows that if we set the penalty parameter µ large enough, we can get a decrease in the objective value of f (w t+1) − f (w t) which is proportional to ∇f (w t) 2. However, this is not the way that the algorithm works. In the algorithm, we only use K devices that are chosen randomly to approximatew t. So, in order to find the E f (w t+1), we use local Lipschitz continuity of the function f. DISPLAYFORM10 where L 0 is the local Lipschitz continuity constant of function f and we have DISPLAYFORM11 Therefore, if we take expectation with respect to the choice of devices in round t we need to bound DISPLAYFORM12 where Q t = E St L 0 w t+1 −w t+1. Note that the expectation is taken over the random choice of devices to update. DISPLAYFORM13 From FORMULA19, we have that DISPLAYFORM14 and DISPLAYFORM15 where the first inequality is a of K devices being chosen randomly to get w t and the last inequality is due to bounded dissimilarity assumption. If we replace these bounds in we get DISPLAYFORM16 Combining,, and and using the notation α = 1 µ we get DISPLAYFORM17 Theorem 3 uses the dissimilarity in Definition 2 to identify sufficient decrease at each iteration for FedProx. Here we provide a corollary characterizing the performance with a more common (though slightly more restrictive) bounded variance assumption. This assumption is commonly employed, e.g., when analyzing methods such as SGD. Corollary 6 (Bounded Variance Equivalence). Let Assumption 1 hold. Then, in the case of bounded variance, i.e., DISPLAYFORM0 Proof. We have, DISPLAYFORM1 With Corollary 6 in place, we can restate the main in Theorem 3 in terms of the bounded variance assumption. Theorem 7 (Non-Convex FedProx Convergence: Bounded Variance). Let the assertions of Theorem 3 hold. In addition, let the iterate w t be such that ∇f (w t) 2 ≥, and let E k ∇F k (w) − ∇f (w) 2 ≤ σ 2 hold instead of the dissimilarity condition. If µ, K and γ in Algorithm 1 are chosen such that DISPLAYFORM2 then at iteration t of Algorithm 1, we have the following expected decrease in the global objective: DISPLAYFORM3 where S t is the set of K devices chosen at iteration t. The proof of Theorem 7 follows from the proof of Theorem 3 by noting the relationship between the bounded variance assumption and the dissimilarity assumption as portrayed by Corollary 6. Corollary 8 (Convergence: Convex Case). Let the assertions of Theorem 3 hold. In addition, let F k (·) be convex and γ = 0, i.e., all the local problems are solved exactly. If 1 B ≤ 0.5 √ K, then we can choose µ ≈ 6LB 2 from which it follows that ρ ≈ 1 24LB 2.Proof. In the convex case, where L − = 0 andμ = µ, if γ = 0, i.e., all subproblems are solved accurately, we can get a decrease proportional to ∇f (w t) 2 if B < √ K. In such a case if we assume 1 << B ≤ 0.5 √ K, then we can write DISPLAYFORM0 In this case, if we choose µ ≈ 6LB 2 we get DISPLAYFORM1 Note that the expectation in FORMULA0 is a conditional expectation conditioned on the previous iterate. Taking expectation of both sides, and telescoping, we have that the number of iterations to at least generate one solution with squared norm of gradient less than is O(DISPLAYFORM2 Remark 9 (Comparison with SGD). Note that FedProx achieves the same asymptotic convergence guarantee as SGD. In other words, under the bounded variance assumption, for small, if we replace B with its upper-bound in Corollary 6 and choose µ large enough, then the iteration complexity of FedProx when the subproblems are solved exactly and DISPLAYFORM0 ), which is the same as SGD BID6. Synthetic data. To generate synthetic data, we follow a similar setup to that described in , additionally imposing heterogeneity among devices. Full details are given in Appendix B.1. In particular, for each device k, we generate synthetic samples DISPLAYFORM0, where the covariance matrix Σ is diagonal with Σ j,j = j −1.2. Each element in the mean vector v k is drawn from N (B k, 1), B k ∼ N (0, β). Therefore, α controls how much local models differ from each other and β controls how much the local data at each device differs from that of other devices. We vary α, β to generate three heterogeneous distributed datasets, Synthetic (α, β), as shown in Figure 1. We also generate one IID dataset by setting the same W, b on all devices and setting X k to follow the same distribution. Our goal is to learn a global W and b. Real data. We also explore five real datasets, their statistics summarized in TAB3 in Appendix B.1. These datasets are curated from prior work in federated learning as well as recent federated learning-related benchmarks (; BID3 . We study two convex models on partitioned MNIST , Federated Extended MNIST BID4 BID3 ) (FEMNIST), and FMNIST*. We study two non-convex models on Sentiment140 BID7 ) FORMULA0 Implementation. We implement FedAvg and FedProx in Tensorflow BID0. See details in Appendix B.2.Setup. For each experiment, we tune the learning rate and ratio of active devices per round on FedAvg. We randomly split the data on each local device into 80% training set and 20% testing set. For each comparison, the devices selected and data read at each round are the same across all runs. We report all metrics based on the global objective f (w). Note that FedAvg (µ = 0) and FedProx (µ ≥ 0) perform the same amount of work at each round when the number of local epochs, E, is the same; we therefore report in terms of rounds rather than FLOPs or wall-clock time. Here we provide full details on the datasets and models used in our experiments. We curate a diverse set of non-synthetic datasets, including those used in prior work on federated learning , and some proposed in LEAF, a benchmark for federated settings BID3. We also create synthetic data to directly test the effect of heterogeneity on convergence, as in Section 5.• Synthetic: We set (α, β)=, (0.5,0.5) and respectively to generate three non-identical distributed datasets (Figure 1). In the IID data, we set the same W, b ∼ N on all devices and X k to follow the same distribution N (v, Σ) where each element in the mean vector v is drawn from N and Σ is diagonal with Σ j,j = j −1.2. For all synthetic datasets, there are 30 devices in total and the number of samples on each device follows a power law.• MNIST: We study image classification of handwritten digits 0-9 in MNIST using multinomial logistic regression. To simulate a heterogeneous setting, we distribute the data among 1000 devices such that each device has samples of only 2 digits and the number of samples per device follows a power law. The input of the model is a flattened 784-dimensional (28 × 28) image, and the output is a class label between 0 and 9.• FEMNIST: We study an image classification problem on the 62-class EMNIST dataset BID4 using multinomial logistic regression. Each device corresponds to a writer of the digits/characters in EMNIST. We call this federated version of EMNIST FEMNIST. The input of the model is a flattened 784-dimensional (28 × 28) image, and the output is a class label between 0 and 61.• Shakespeare: This is a dataset built from The Complete Works of William Shakespeare . Each speaking role in a play represents a different device. We use a two layer LSTM classifier containing 100 hidden units with a 8D embedding layer. The task is next character prediction and there are 80 classes of characters in total. The model takes as input a sequence of 80 characters, embeds each of the character into a learned 8 dimensional space and outputs one character per training sample after 2 LSTM layers and a densely-connected layer.• Sent140: In non-convex settings, we consider a text sentiment analysis task on tweets from Sentiment140 BID7 ) (Sent140) with a two layer LSTM binary classifier containing 256 hidden units with pretrained 300D GloVe embedding . Each twitter account corresponds to a device. The model takes as input a sequence of 25 characters, embeds each of the character into a 300 dimensional space by looking up Glove and outputs one character per training sample after 2 LSTM layers and a densely-connected layer. • FEMNIST*: We generate FEMNIST* by subsampling 26 lower case characters from FEMNIST and distributing only 20 classes to each device. There are 200 devices in total. The model is the same as the one used on FEMNIST. We report the total number of devices, samples, and the mean and standard deviation of samples per device of real federated datasets in TAB3. (Implementation) In order to draw a fair comparison with FedAvg, we use SGD as a local solver for FedProx, and adopt a slightly different device sampling scheme than that in Algorithms FedAvg and 1: sampling devices uniformly and averaging updates with weights proportional to the number of local data points (as originally proposed in ). While this sampling scheme is not supported by our analysis, we observe similar relative behavior of FedProx vs. FedAvg whether or not it is employed. Interestingly, we also observe that the sampling scheme proposed herein in more stable performance for both methods (see Appendix B.4, Figure 10). This suggests an added benefit of the proposed framework.(Machines) We simulate the federated learning setup (1 server and N devices) on a commodity machine with 2 Intel R Xeon R E5-2650 v4 CPUs and 8 NVidia R 1080Ti GPUs.(Hyperparameters) For each dataset, we tune the ratio of active clients per round from {0.01, 0.05, 0.1} on FedAvg. For synthetic datasets, roughly 10% of the devices are active at each round. For MNIST, FEMNIST, Shakespeare, Sent140 and FEMNIST*, the number of active devices (K) are 1%, 5%, 10%, 1% and 5% respectively. We also do a grid search on the learning rate based on FedAvg. We do not decay the learning rate through all rounds. For all synthetic data experiments, the learning rate is 0.01. For MNIST, FEMNIST, Shakespeare, Sent140 and FEMNIST*, we use the learning rates of 0.03, 0.003, 0.8, 0.3 and 0.003. We use a batch size of 10 for all experiments.(Libraries) All code is implemented in Tensorflow BID0 Version 1.10.1. Please see github.com/litian96/FedProx for full details. We explore the effect of E in Figure 4. For each dataset, we set E to be 1, 20, and 50 while keeping µ = 0 (FedProx reduces to FedAvg in this case) and show the convergence in terms of the training loss. We see that large E leads to divergence or instability on MNIST and Shakespeare. On FEMNIST and Sent140, nevertheless, larger E speeds up the convergence. Based on drawn from Figure 1, we hypothesize this is due to the fact that the data distributed across devices after partitioning FEMNIST and Sent140 lack significant heterogeneity. We validate this hypothesis by observing instability on FEMNIST*, which is a skewed variant of the FEMNIST dataset. We consider the effect of µ on convergence in Figure 5. For each experiment, in the case of E = 50, we compare the between µ = 0 and the best µ. For three out of the four datasets (all but Sent140) we observe that the appropriate µ can increase the stability for unstable methods and can force divergent methods to converge. Finally, in Figure 6, we demonstrate that our B-local dissimilarity measurement in Definition 2 captures the heterogeneity of datasets and is therefore an appropriate proxy of performance. In particular, we track the variance of gradients on each device, DISPLAYFORM0, which is lower bounded by B (see Bounded Variance Equivalence Corollary 6). We observe that the dissimilarity metric is consistent with the training loss. Therefore, smaller dissimilarity indicates better convergence, which can be enforced by setting µ appropriately. Full tracking B (for all experiments performed) are provided in Appendix B.3.We present testing accuracy, training loss and dissimilarity measurements of all the experiments in FIG4, Figure 8 and Figure 9. We show the training loss, testing accuracy and dissimilarity measurement of FedProx using two different device sampling schemes in Figure 10. We show a simple adaptive heuristic of setting µ on four synthetic datasets in Figure 11. Two aspects of the proposed work: our framework, FedProx, and analysis tool, the bounded dissimilarity assumption, have been utilized throughout the optimization literature-though often with very different motivations. For completeness, we provide a discussion below on our relation to these prior works. Figure 9. Training loss, testing accuracy and dissimilarity measurement for experiments in Figure 5 Proximal term. We note here a connection to elastic averaging SGD (EASGD) , which was proposed as a way to train deep networks in the data center setting, and uses a similar proximal term in its objective. While the intuition is similar to EASGD (this term helps to prevent large deviations on each device/machine), EASGD employs a more complex moving average to update parameters, is limited to using SGD as a local solver, and has only been analyzed Figure 10. Differences between two sampling schemes in terms of training loss, testing accuracy and dissimilarity measurement. Sampling devices with a probability proportional to the number of local data points and then simply averaging local models performs slightly better than uniformly sampling devices and averaging the local models with weights proportional to the number of local data points. Under either sampling scheme, the settings with µ = 1 demonstrate more stable performance than settings with µ = 0.for simple quadratic problems. The proximal term we introduce has also been explored in previous optimization literature with very different purposes, such as , to speed up (mini-batch) SGD training on a single machine. Li et al. (2014b) also employs a similar proximal term for efficient SGD training both in a single machine and distributed settings, but their analysis is limited to a single machine setting with different assumptions (e.g., IID data and solving the subproblem exactly at each round). DANE also includes a proximal term in the local objective function. However, due to the inexact estimation of full gradients (i.e., ∇φ(w (t−1) ) in (, Eq )) with device subsampling schemes and the staleness of the gradient correction term (, Eq ) in local updating methods, it is not directly applicable to our setting and performs worse on heterogeneous datasets (see Figure 12).Bounded dissimilarity assumption. The bounded dissimilarity assumption has appeared in different forms, for example in . In , the bounded similarity assumption is used in context of asserting gradient diversity and quantifying the benefit in terms of scaling of the mean square error for mini-batch SGD for data which is i.i.d. In , the authors use a similar assumption, called strong growth condition, which is a stronger version of Assumption 1 with = 0. They prove that some interesting practical problems satisfy such a condition. They also use this assumption to prove better convergence rates for SGD with constant step-size. Note that this is different with our approach as the algorithm that we are analyzing is not SGD and our analysis is different in spite of the similarity in the assumptions.

We introduce FedProx, a framework to tackle statistical heterogeneity in federated settings with convergence guarantees and improved robustness and stability.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Referential games offer a grounded learning environment for neural agents which accounts for the fact that language is functionally used to communicate. However, they do not take into account a second constraint considered to be fundamental for the shape of human language: that it must be learnable by new language learners and thus has to overcome a transmission bottleneck. In this work, we insert such a bottleneck in a referential game, by introducing a changing population of agents in which new agents learn by playing with more experienced agents. We show that mere cultural transmission in a substantial improvement in language efficiency and communicative success, measured in convergence speed, degree of structure in the emerged languages and within-population consistency of the language. However, as our core contribution, we show that the optimal situation is to co-evolve language and agents. When we allow the agent population to evolve through genotypical evolution, we achieve across the board improvements on all considered metrics. These stress that for language emergence studies cultural evolution is important, but also the suitability of the architecture itself should be considered. Human languages show a remarkable degree of structure and complexity, and how such a complex system can have emerged is still an open question. One concept frequently named in the context of language evolution is cultural evolution. Unlike animal languages, which are taken to be mostly innate, human languages must be re-acquired by each individual BID29 BID10. This pressures them to fit two constraints that govern their cross-generational transmission: They must be learnable by new language users, and they must allow effective communication between proficient language users (see, e.g. BID31 .In the recent past, computational studies of language emergence using referential games (see Section 2.1 for a review) has received a new wave of attention. These studies are motivated by the second constraint, that language is used to communicate. The first constraint, on the other hand, is in this framework not considered: language is not transmitted from agent to agent and there is thus no need for agents to develop languages that would survive a transmission bottleneck. 1 In this work, we introduce a transmission bottleneck in a population of agents playing referential games, implicitly modelling cultural evolution. However, merely adding a transmission bottleneck is not enough. Since the types of language that may emerge through passing this bottleneck are not just dependent on the existence of a bottleneck, but also on the shape of the bottleneck, which is determined by the biases of the architecture of the agents playing the game (their genotypical design). If the genotypical design of those agents is not suitable to solve this task through communication, they will -at best -converge to a language that doesn't allow for effective communication or is difficult to learn for every new agent or -at worst -not converge to an appropriate culturally transmittable language at all. In this work, we therefore study the co-evolution of language and architecture in a referential games. To this end, we introduce the Language Transmission Engine that allows to model both cultural and genetic evolution in a population of agents. We demonstrate that the emerging languages ben-efit from including cultural transmission as well as genetic evolution, but the best are achieved when both types of evolution are included and languages and agents can co-evolve.2 Related Work Much work has been done on the emergence of language in artificial agents and investigating its subsequent structure, compositionality and morphosyntax BID15 BID17. The original computer simulations dealt with logic and symbolic representations BID15 BID3, but with the advent of modern deep learning methods and sequence-to-sequence models BID33, there has been a renewed interest in simulating the emergence of language through neural network agents (i.a. BID21 BID8 . In the exploration of language emergence, different training approaches and tasks have been proposed to encourage agents to learn and develop communication. These tasks are commonly set up in an end-to-end setting where reinforcement learning can be applied. This is often a two-player referential game where one agent must communicate the information it has access to (typically an image), while the other must guess it out of a lineup BID6 BID21. BID25 and BID2 find that structure and compositionalility can arise in emerged languages in such setups; BID19 show that'natural' language does not arise naturally and has to be incentivised by imposing specific restrictions on games and agents. The evolution of human language is a well-studied but still poorly understood topic. One particular open question concerns the relation between two different evolutionary processes: genetic evolution of the agents in the population and cultural evolution of the language itself BID7. BID3 assert that the question of genetic versus cultural evolution ultimately arises from three distinct but interacting adaptive systems: individual learning, cultural transmission, and genetic evolution. Cultural transmission is thought to enforce structure and compression to languages, since a language must be used and learned by all individuals of the culture in which it resides and at the same time be suitable for a variety of tasks. BID18 define those two pressures as compressibility and expressivity and find that structure arises from the trade-off between these pressures in generated languages. The importance of cultural evolution for the emergence of structure is supported by a number of artificial language learning studies (e.g. BID30 and computational studies using the Iterated Learning paradigm, in which agents learn a language by observing the output produced by another agent from the previous 'generation' (e.g. BID13 BID16 BID18 . An alternative way of imposing cultural pressures on agents, is by simulating a large population of them and pairing agents randomly to solve a communicative game BID4 . This approach is more naturally aligned with cultural pressures in humans (see e.g. BID34 and is the one we use in this paper. While there is much controversy about the selection pressures under which the fundamental traits underlying the human ability to learn and use language evolved in other humans, that genetic evolution played an essential role in endowing humans with the capabilities to learn and use language is generally undebated. Pre-modern humans, for instance, did not have the ability to speak or understand complex structures BID7 .There are several approaches to simulate genetic evolution of neural network agents. Neural Architectural Search (NAS) focuses on searching the architecture space of the networks, unlike many traditional evolutionary techniques which often include parameter weights in their search space. Some of the earlier techniques such as NEAT gained considerable traction as a sound way of doing topology search using biologically inspired concepts BID32. NAS methods however have mostly reverted to optimising solely the neural architecture and using gradient based methods such as SGD for weight optimisation due to the large parameter space of modern architectures (see, e.g., BID5 for a survey).More recently, state-of-the-art one-shot search techniques such as ENAS (Efficient Neural Architecture Search) and DARTS (Differentiable Architecture Search) have allowed to bring a gradientbased approach to NAS through the use of intelligent weight-sharing schemes BID24 ).In this work, we use the DARTS search space, which is constrained but still obtained state-of-the-art performance on benchmark natural language tasks BID23. We study language emergence in a referential game inspired by the signalling games proposed by BID22. In this game, one agent (called the sender) observes an image and generates a discrete message. The other agent, the receiver of the message, uses the message to select the right image from a set of images containing both the sender image and several distractor images. Since the information shown to the sender agent is crucial to the receivers success, this setup urges the two agents to come up with a communication protocol that conveys the right information. Formally, our referential game is similar to BID8:We use z = 512, and n = 3 and train agents with Gumbel-Softmax BID11 ) based on task-success. We introduce both cultural and genetic evolution to this game through a process that we call the Language Transmission Engine (LTE), which is depicted in FIG0. 2 Similar to , we create a population of communicating agents. In every training iteration, two random agents are sampled to play the game. This forces the agents to adopt a simpler language naturally: to succeed they must be able to communicate or understand all opposing agents. In our setup, agents are either sender or receiver, they do not switch roles during their lifetime. To model cultural evolution in the LTE, we periodically replace agents in the population with newly initialised agents. Cultural evolution is implicitly modelled in this setup, as new agents have to learn to communicate with agents that already master the task. Following BID4, we experiment with three different methods to select the agents that are replaced: randomly (no selection pressure), replacing the oldest agents or replacing the agents with the lowest fitness (as defined in Section 3.3). We call these setups cu-random, cu-age and cu-best, respectively. To model genetic evolution, rather than periodically replacing agents with randomly initialised new agents, we instead mutate the most successful agents and replace the worst agents with variations of the best agents, as outlined in Section 3.2.2. Note that cultural evolution is still implicitly modelled in this setup, as new agents still have to learn to communicate with older agents. Therefore, we call this setup with the term co-evolution. Culling We refer to the selection process and subsequent mutation or re-initialisation step as culling. In biology, culling is the process of artificially removing organisms from a group to promote certain characteristics, so, in this case, culling consists of removing a subset of the worst agents and replacing them with variations of the best architecture. The proportion of agents from each population selected to be mutated is determined by the culling rate ↵, where ↵ 2. The culling interval l defines the number of iterations between culling steps. A formalisation of the LTE can be found in appendix A.1. We base potential mutations on the RNN cell search space DARTS, defined by BID24. This space includes recurrent cells with up to N nodes, where each node n 1, n 2,..., n N can take the output of any preceding nodes including n 0, which represents the cell's input. All potential connections are modulated by an activation function, which can be the identity function, Tanh, Sigmoid or ReLU. Following BID24 and , we enhance each operation with a highway bypass BID35 and the average of all intermediate nodes is treated as the cell output. To sample the initial model, we sample a random cell with a single node (N = 1). As this node must necessarily be connected to the input, the only variation stems from the possible activation functions applied to the output of n 1, ing in four possible starting configurations. We set a node cap of N = 8. We mutate cells by randomly sampling an architecture which is one edit step away from the previous architecture. Edit steps are uniformly sampled from i) changing an incoming connection, ii) changing an output operation or iii) adding a new node; the mutation location is uniformly sample from all possible mutations. 3 Note that while we use the DARTS search space to define potential mutations, contrary to BID24, we do not use differentation to sample new architectures based on a selection criterion. The fitness criterion that we use in both the cu-best and co-evolution setup is based on task performance. However, rather than considering agents' performance right before the culling step, we consider the age of the youngest agent in the population (defined in terms of number of batches that it was trained) and for every agent compute their performance up until when they had DISPLAYFORM0 where T A = min a2A T (a) is the age T (a) of the youngest agent in the population, and L(a t j) is the loss of agent a j at time step t. This fitness criterion is not biased towards older agents, that have seem already more data and have simply converged more. It is thus not only considering task performance but also the speed at which this performance is reached. We test the LTE framework on a compositionally defined image dataset, using a range of different selection mechanisms. In all our experiments, we use a modified version of the Shapes dataset BID0, which consists of 30 by 30 pixel images of 2D objects, characterised by shape (circle, square, triangle), colour (red, green, blue), and size (small, big). While every image has a unique symbolic description -consisting of the shape, colour and size of the object and its horizontal and vertical position in a 3x3 grid -one symbolic representation maps to multiple images, that differ in terms of exact pixels and object location. We use 80k, 8k, 40k images for train, validation and test sets, respectively. Some example images are depicted in FIG1.We pre-train a CNN feature extractor for the images in a two-agent setting of the task (see Appendix A.4 for more details). For our co-evolution experiments, we use the DARTS search space as described above. For all cultural evolution approaches, we use an LSTM BID9 for both the sender and receiver architectures (see Appendix A.3 for more details). Unless otherwise specified, we use the same sizes and hyper-parameters for all models. The sender and receiver models have a hidden size of 64 for the recurrent layer and an embedding layer of size 64. Further, we use a vocabulary size V of 4, with an additional bound token serving as the indicator for beginning and end- of-sequence. We limit the maximum length of a sentence L to 5.We back-propagate gradients through the discrete step outputs (message) of the sender by using the Straight-Through (ST) Gumbel-Softmax Estimator BID12. We run all experiments with a fixed temperature ⌧ = 1.2. We use the default Pytorch BID26 Adam optimiser with a learning rate of 0.001 and a batch-size of 1024. Note that the optimiser is reset for every batch. For all multi-agent experiments we use a population size of 16 senders and 16 receivers. The culling rate ↵ is set to 0.25 or four agents, and we cull (re-initialise or mutate) every l = 5k iterations. We run the experiments for a total of I = 500k iterations, and evaluate the populations before each culling step. We use an range of metrics to evaluate both the population of agents and the emerging languages. Jaccard Similarity We measure the consistency of the emerged languages throughout the population using Jaccard Similarity, which is defined as the ratio between the size of the intersection and the union of two sets. We sample 200 messages per input image for each possible sender-receiver pair and average the Jaccard Similarity of the samples over the population. A high Jaccard Similarity between two messages is an indication that the same tokens are used in both messages. We compute how similar the messages that different agents emit for the same inputs by looking at all possible (sender, message) pairs for one input and assess whether they are the same. This metric is 1 when all agents always emit the same messages for the same inputs. We compute the average number of unique messages generated by each sender in the population. An intuitive reference point for this metric is the number of images with distinct symbolic representations. If agents generate more messages than expected by this reference point, this demonstrates that they use multiple messages for the images that are -from a task perspective -identical. A smaller number of unique messages, on the other hand, indicates that the agent is using a simpler language which is underspecified compared to the symbolic description of the image. Topographic Similarity Topographic similarity, used in a similar context by, represents the similarity between the meaning space (defined by the symbolic representations) and the signal space (the messages send by an agent). It is defined as the correlation between the distances between pairs in meaning space and the distances between the corresponding messages in the signal space. We compute the topographic similarity for an agent by sampling 5,000 pairs of symbolic inputs and corresponding messages and compute the Pearson's ⇢ correlation between the cosine similarity of the one-hot encoded symbolic input pairs and the cosine similarity of the one-hot encoded message pairs. Average Population Convergence To estimate the speed of learning of the agents in the population, estimate the average population convergence. For each agent, at each point in time, this is defined as the agents average performance from the time it was born until it had the age of the current youngest agent in the population (analogous to the fitness criterion defined in Section 3.3). To get the average population convergence, we average those values for all agents in the population. Average Agent Entropy We compute the average certainty of sender agents in their generation process by computing and averaging their entropy during generation. We now present a detailed comparison of our cultural and co-evolution setups. For each approach, we averaged over four random seeds, the error bars in all plots represent the standard deviation across these four runs. To analyse the evolution of both agents and languages, we consider the development of all previously outlined metrics over time. We then test the best converged languages and architectures in a single sender-receiver setup, to assess the impact of cultural and genetic evolution more independently. In these experiments, we compare also directly to a single sender-receiver baseline, which is impossible for most of the metrics we consider in this paper. Finally, we briefly consider the emerged architectures from a qualitative perspective. We first confirm that all setups in fact converge to a solution to the task. As can be seen in FIG2, all populations converge to a (close to perfect) solution to the game. The cu-age approach slightly outperforms the other approaches, with a accuracy that surpasses the 95% accuracy mark. Note that, due to the ever changing population, the accuracy at any point in time is an average of both'children' and'adults', that communicate with different members of the population. To assess the behaviour of the agents over time, we monitor their average message entropy convergence speed. As can be seen in FIG3, the co-evolution setup in the lowest average entropy scores, the messages that they assign to one particular image will thus have lower variation than in the other setups. Of the cultural evolution setups, the lowest entropy score is achieved in the cu-best setup. FIG4 shows the average population convergence over time. Also in this case, we observe a clear difference between cultural evolution only and co-evolution, with an immediately much lower convergence time for co-evolution and a slightly downward trending curve. To check the consistencies of languages within a population, we compare the Jaccard Similarity and the Average Proportion of Unique Matches, which we plot in Figure 6. This shows that, compared to cultural evolution only, not only are the messages in co-evolution more similar across agents (higher Jaccard Similarity), but also that agents are considerably more aligned with respect to the same inputs (less unique matches).To assess the level of structure of the emerged languages, we plot the average Topographic Similarity and the Average Number of Unique Messages generated by all senders (Figure 7). The co-evolution condition again outperforms all cultural only conditions, with a simpler language (the number of the unique messages closer to the symbolic reference point) that is structurally more sim- Figure 6: Average Jaccard Similarity and proportion of message matches for all cultural transmission modes and evolution ilar to the symbolic representation of the input (higher Topographical Similarity). In Figure 8 we show the co-evolution of an agent and a sample of its language during three selected iterations in the co-evolution setup. Strikingly, the best sender architecture does not evolve from its original form, which could point towards the limitations of of our search strategy and space. On the contrary, the receiver goes through quite some evolution steps and converges into a significantly more complex architecture than its original form. We observe a unification of language throughout evolution in Figure 8, which is also supported by Figure 7. The population of senders starts out 11 different unique messages and ends with only two to describe the same input image. We will leave more detailed analysis of the evolved architectures for future work. With a series of experiments we test the a priori suitability of the evolved languages and agents for the task at hand, by monitoring the accuracy of new agents that are paired with converged agents and train them from scratch. We focus, in particular, on training receivers with a frozen sender from different setups, which allows us to assess 1) whether cultural evolution made languages evolve to be more easily picked up by new agents 2) whether the genetic evolution made architectures converge more quickly when faced with this task. We compare the accuracy development of: Figure 7: Average Number of Unique Messages and Topographic Similarity for all cultural evolution modes and co-evolution. For comparison, we also plot the number of unique messages for a symbolic solution that fully encodes all relevant features of the image (since we have three possible shapes and colours, two possible sizes, and a 3 ⇥ 3 grid of possible positions, this symbolic reference solution has 3 ⇥ 3 ⇥ 2 ⇥ 9 = 162 distinct messages.• An LSTM receiver trained with a frozen sender taken from cu-best;• An evolved receiver trained with a frozen evolved sender. For both these experiments, we compare with two baselines:• The performance of a receiver agent trained from scratch along with a receiver agent that has either the cu architecture or the evolved co architecture (cu-baseline and co-baseline, respectively);• The performance of an agent trained with an agent that is pretrained in the single agent setup, with either the cu architecture or an evolved architecture (cu-baseline-pretrained and co-baseline-pretrained).Each experiment is run 10 times, keeping the same frozen agent. The confirm cultural evolution contributes to the learnability and suitability of emerging languages: the cu-best accuracy (green line) converges substantially quicker and is substantially higher than the cu-baseline-pretrained accuracy (orange line). Selective pressure on the Figure 8: Evolution of the best sender and receiver architecture according to convergence, and the evolution of the population's message description of the same input through iterations. The bold messages represent the message outputted by the best sender whose architecture is pictured above. The count of each message represents the number of agents in the population which uttered this exact sequence. language appears to be important: the ing languages are only easier to learn in the cu-best setup. 4 In addition, they show that the agents benefit also from the genetic evolution: the best accuracies are achieved in the co-evolution setup (red line). The difference between the cu-baseline (blue) and the co-baseline (brown) further shows that even if the evolved architectures are trained from scratch, they perform much better than a baseline model trained from scratch. The difference between the co-baseline-pretrained (only genetic evolution, purple line) and the co-evolution of agents and language line (red line) illustrates that genetic evolution alone is not enough: while a new evolved receiver certainly benefits from learning from a (from scratch) pretrained evolved sender, without the cultural transmission pressure, it's performance is still substantially below a receiver that learns from an evolved sender whose language was evolved as well. In this paper, we introduced a language transmission bottleneck in a referential game, where new agents have to learn the language by playing with more experienced agents. To overcome such bottleneck, we enabled both the cultural evolution of language and the genetic evolution of agents, using a new Language Transmission Engine. Us- ing a battery of metrics, we monitored their respective impact on communication efficiency, degree of linguistic structure and intra-population language homogeneity. While we could find important differences in between cultural evolution strategies, it is when we included genetic evolution that agents scored best. In a second experiment, we paired new agents with evolved languages and agents and again confirmed that, while cultural evolution makes a language easier to learn, coevolution leads to the best communication. In future research, we would like to apply the Language Transmission Engine on new, more complex tasks and further increase our understanding of the properties of the emerged languages and architectures. Additionally, we would like to investigate other neuro-evolution techniques and apply them on different search spaces.

We enable both the cultural evolution of language and the genetic evolution of agents in a referential game, using a new Language Transmission Engine.

Represent the short summary for finding the corresponding scientific article (long)

Our paper analyses the tremendous representational power of networks especially with 'skip connections', which may be used as a method  for better generalization.