Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems Reinforcement learning has recently made exciting progress in many domains, including Atari games, the ancient Chinese board game, Go, and complex continuous control tasks involving locomotion BID17 BID26 BID12. While most reinforcement learning paradigms focus on single agents acting in a static environment (or against themselves in the case of Go), real-world agents often compete or cooperate with other agents in a dynamically shifting environment. In order to learn effectively in multi-agent environments, agents must not only learn the dynamics of their environment, but also those of the other learning agents present. To this end, several approaches for multi-agent reinforcement learning have been developed. The simplest approach is to train each agent independently to maximize their individual reward, while treating other agents as part of the environment. However, this approach violates the basic assumption underlying reinforcement learning, that the environment should be stationary and Markovian. Any single agent's environment is dynamic and nonstationary due to other agents' changing policies. As such, standard algorithms developed for stationary Markov decision processes fail. At the other end of the spectrum, all agents can be collectively modeled as a single-agent whose action space is the joint action space of all agents BID2. While allowing coordinated behaviors across agents, this approach is not scalable due to the action space size increasing exponentially with the number of agents. It also demands a high degree of communication during execution, as the central policy must collect observations from and distribute actions to the individual agents. In real-world settings, this demand can be problematic. Recent work BID20 attempts to combine the strengths of these two approaches. In particular, a critic (or a number of critics) is centrally learned with information from all agents. The actors, however, receive information only from their corresponding agents. Thus, during testing, executing the policies does not require the knowledge of other agents' actions. This paradigm circumvents the challenge of non-Markovian and non-stationary environments during learning. Despite those progresses, however, algorithms for multi-agent reinforcement learning are still far from being scalable (to a larger number of agents) and being generically applicable to environments and tasks that are co-operative (sharing a global reward), competitive, or mixed. Our approach extends these prior works in several directions. The main idea is to centrally learn a critic with an attention mechanism. The intuition behind our idea is that in many real-world environ-ments, it is beneficial for agents to know what other agents it should pay attention to. For example, a soccer defender needs to pay attention to attackers in their vicinity as well as the player with the ball, while she/he rarely needs to pay attention to the opposing team's goalie. The specific attackers that the defender is paying attention to can change at different parts of the game, depending on the formation and strategy of the opponent. A typical centralized approach to multi-agent reinforcement learning does not take these dynamics into account, instead simply considering all agents at all timepoints. Our attention mechanism is able to dynamically select which agents to attend to at each time point, improving performance in multi-agent domains with complex interactions. The proposed approach has an input space linearly increasing with respect to the number of agents, as opposed to the quadratic increase in a previous approach BID20. It also works well in co-operative, competitive, and mixed environments, exceeding the capability of some prior work that focuses only on co-operative environments.We have validated our approach on two simulated environments and tasks. We plan to release the code for both the model and the environments after the reviewing period ends. The rest of the paper is organized as follows. In section 2, we discuss related work, followed by a detailed description of our approach in section 3. We report experimental studies in section 4 and conclude in section 5. Multi-Agent Reinforcement Learning (MARL) is a long studied problem BID2. Topics within MARL are diverse, ranging from learning communication between cooperative agents (; BID4 to algorithms for optimal play in competitive settings BID19, though, until recently, they have been focused on simple gridworld environments with tabular learning methods. As deep learning based approaches to reinforcement learning have grown more popular, they have, naturally, been applied to the MARL setting BID32 BID9, allowing multi-agent learning in high-dimensional/continuous state spaces; however, naive applications of Deep RL methods to MARL naturally encounter some limitations, such as nonstationarity of the environment from the perspective of individual agents BID6 BID20, lack of coordination/communication in cooperative settings BID29 BID23 BID20 BID5, credit assignment in cooperative settings with global rewards BID25 BID30, and the failure to take opponent strategies into account when learning agent policies BID11 .Most relevant to this work are recent, non-attention approaches that propose an actor-critic framework consisting of centralized training with decentralized execution BID20, as well as some approaches that utilize attention in a fully centralized multi-agent setting BID3 BID14 . BID20 investigate the challenges of multiagent learning in mixed reward environments BID2 . They propose an actor-critic method that uses separate centralized critics for each agent which take in all other agents' actions and observations as input, while training policies that are conditioned only on local information. This practice reduces the non-stationarity of multi-agent environments, as considering the actions of other agents to be part of the environment makes the state transition dynamics stable from the perspective of one agent. In practice, these ideas greatly stabilize learning, due to reduced variance in the value function estimates. Similarly introduce a centralized critic for cooperative settings with shared rewards. Their method incorporates a "counterfactual baseline" for calculating the advantage function which is able to marginalize a single agent's actions while keeping others fixed. This method allows for complex multi-agent credit assignment, as the advantage function only encourages actions that directly influence an agent's rewards. Attention models have recently emerged as a successful approach to intelligently selecting contextual information, with applications in computer vision BID0 BID21, natural language processing (; BID1 BID18, and reinforcement learning BID24 .In a similar vein, BID14 proposed an attention-based actor-critic algorithm for MARL. This work follows the alternative paradigm of centralizing policies while keeping the critics decentralized. Their focus is on learning an attention model for sharing information between the policies. As such, this approach is complementary to ours, and a combination of both approaches could yield further performance benefits in cases where centralized policies are desirable. Our proposed approach is more flexible than the aformentioned approaches for MARL. Our algorithm is able to train policies in environments with any reward setup, different action spaces for each agent, a variance-reducing baseline that only marginalizes the relevant agent's actions, and with a set of centralized critics that dynamically attend to the relevant information for each agent at each time point. As such, our approach is more scalable to the number of agents, and is more broadly applicable to different types of environments. We start by introducing the necessary notation and basic building blocks for our approach. We then describe our ideas in detail. We consider the framework of Markov Games BID19, which is a multi-agent extension of Markov Decision Processes. They are defined by a set of states, S, action sets for each of N agents, A 1, ..., A N, a state transition function, T : S × A 1 × ... × A N → P (S), which defines the probability distribution over possible next states, given the current state and actions for each agent, and a reward function for each agent that also depends on the global state and actions of all agents, DISPLAYFORM0 We will specifically be considering a partially observable variant in which an agent, i receives an observation, o i, which contains partial information from the global state, s ∈ S. Each agent learns a policy, π i: O i → P (A i) which maps each agent's observation to a distribution over it's set of actions. The agents aim to learn a policy that maximizes their expected discounted returns, DISPLAYFORM1, where γ ∈ is the discount factor that determines how much the policy favors immediate reward over long-term gain. Policy Gradients Policy gradient techniques BID31 ) aim to estimate the gradient of an agent's expected returns with respect to the parameters of its policy. This gradient estimate takes the following form: DISPLAYFORM2 Actor-Critic and Soft Actor-Critic The term ∞ t =t γ t −t r t (s t, a t) in the policy gradient estimator leads to high variance, as these returns can vary drastically between episodes. Actor-critic methods BID16 aim to ameliorate this issue by using a function approximation of the expected returns, and replacing the original return term in the policy gradient estimator with this function. One specific instance of actor-critic methods learns a function to estimate expected discounted returns, given a state and action, Q ψ (s t, a t) = E[DISPLAYFORM3, learned through temporal-difference learning by minimizing the regression loss: DISPLAYFORM4 where Qψ is the target Q-value function. To encourage exploration and avoid converging to non-optimal deterministic policies, recent approaches of maximum entropy reinforcement learning learn a soft value function by modifying the policy gradient to incorporate an entropy term BID10 : DISPLAYFORM5 where b(s) is a state-dependent baseline (for the Q-value function). The loss function for temporaldifference learning of the value function is also revised accordingly with a new target: DISPLAYFORM6 While an estimate of the value function V φ (s) can be used a baseline, we provide an alternative that further reduces variance and addresses credit assignment in the multi-agent setting in section 3.2. The main idea behind our multi-agent learning approach is to learn the critic for each agent by selectively paying attention to other agents' actions. This is the same paradigm of training critics centrally (to overcome the challenge of non-stationary non-Markovian environments) and executing learned policies distributedly. FIG0 illustrates the main components of our approach. Attention The attention mechanism functions in a manner similar to a differentiable key-value memory model BID24. Intuitively, each agent queries the other agents for information about their observations and actions and incorporates that information into the estimate of its value function. This paradigm was chosen, in contrast to other attention-based approaches, as it doesn't make any assumptions about the temporal or spatial locality of the inputs, as opposed to approaches taken in the natural language processing and computer vision fields. DISPLAYFORM0 where f i is a two-layer multi-layer perceptron (MLP), while g i is a one-layer MLP embedding function. The contribution from other agents, x i, is a weighted sum of each agent's value: DISPLAYFORM1 where the value, v j is a function of agent j's embedding, encoded with an embedding function and then linearly transformed by a shared matrix V. h is an element-wise nonlinearity (we have used leaky ReLU).The attention weight α j compares the embedding e j with e i = g i (o i, a i), using a bilinear mapping (ie, the query-key system) and passes the similarity value between these two embeddings into a softmax DISPLAYFORM2 where W q transforms e i into a "query" and W k transforms e j into a "key". The matching is then scaled by the dimensionality of these two matrices to prevent vanishing gradients .In our experiments, we have used multiple attention heads . In this case, each head, using a separate set of parameters (W k, W q, V), gives rise to an aggregated contribution from all other agents to the agent i and we simply concatenate the contributions from all heads as a single vector. Crucially, each head can focus on a different weighted mixture of agents. Note that the weights for extracting selectors, keys, and values are shared across all agents, which encourages a common embedding space. The sharing of critic parameters between agents is possible, even in adversarial settings, because multi-agent value-function approximation is, essentially, a multi-task regression problem. This method can easily be extended to include additional information, beyond local observations and actions, at training time, including the global state if it is available, simply by adding additional encoders, e. (We do not consider this case in our experiments, however, as our approach is effective in combining local observations to predict expected returns in environments where the global state may not be available).Learning with Attentive Critics All critics are updated together to minimize a joint regression loss function, due to the parameter sharing: DISPLAYFORM3, where DISPLAYFORM4 whereψ andθ are the parameters of the target critics and target policies respectively. Note that Q ψ i, the action-value estimate for agent i, receives observations and actions for all agents. α is the temperature parameter determining the balance between maximizing entropy and rewards. The individual policies are updated with the following gradient: DISPLAYFORM5 where b(o, a \i) is the multi-agent baseline used to calculate the advantage function decribed in the following section. Note that we are sampling all actions, a, from all agents' current policies in order to calculate the gradient estimate for agent i, unlike in the MADDPG algorithm BID20, where the other agents' actions are sampled from the replay buffer, potentially causing overgeneralization where agents fail to coordinate based on their current policies. Full training details and hyperparameters can be found in the appendix 6.1. As shown in, an advantage function using a baseline that only marginalizes out the actions of the given agent from Q ψ i (o, a), can help solve the multi-agent credit assignment problem. In other words, by comparing the value of a specific action to the value of the average action for the agent, with all other agents fixed, we can learn whether said action will cause an increase in expected return or whether any increase in reward is attributed to the actions of other agents. The form of this advantage function is shown below: DISPLAYFORM0 Using our attention mechanism, we can implement a more general and flexible form of a multiagent baseline that, unlike the advantage function proposed in, doesn't assume the same action space for each agent, doesn't require a global reward, and attends dynamically to other agents, as in our Q-function. This is made simple by the natural decomposition of an agents encoding, e i, and the weighted sum of encodings of other agents, x i, in our attention model. Concretely, in the case of discrete policies, we can calculate our baseline in a single forward pass by outputting the expected return Q i (o, (a i, a \i)) for every possible action, a i ∈ A i, that agent i can take. We can then calculate the expectation exactly: DISPLAYFORM1 In order to do so, we must remove a i from the input of Q i, and output a value for every action. We add an observation-encoder, e i = g o i (o i), for each agent, using these encodings in place of the e i = g i (o i, a i) described above, and modify f i such that it outputs a value for each possible action, rather than the single input action. In the case of continuous policies, we do not need to add any parameters, as we can simply estimate the expectation in Equation 9 by sampling actions from our policy and averaging their Q-values, though, this comes at the cost of multiple expensive passes through the network. 4.1 SETUP (a) Cooperative Treasure Collection. The small grey agents are "hunters" who collect the colored treasure, and deposit them with the correctly colored large "bank" agents.(b) Rover-Tower. Each grey "Tower" is paired with a "Rover" and a destination (color of rover corresponds to its destination). Their goal is to communicate with the "Rover" such that it moves toward the destination. We construct two environments that test various capabilities of our approach (MAAC) and baselines. We investigate in two main directions. First, we study the scalability of different methods as the number of agents grows. We hypothesize that the current approach of concatenating all agents' observations (often used as a global state to be shared among agents) and actions in order to centralize critics does not scale well. To this end, we implement a cooperative environment, Cooperative Treasure Collection, with shared rewards where we can vary the total number of agents. The experimental in sec 4.3 validate our claim. Secondly, we want to evaluate each method's ability to attend to information relevant to rewards. Moreover, the relevance (to rewards) can dynamically change during an episode. This is analogous to real-life tasks such as the soccer example presented earlier. To this end, we implement a Rover-Tower task environment where randomly paired agents communicate information and coordinate. The two environments are implemented in the multi-agent particle environment framework 1 introduced by BID23, and extended by BID20. We found this framework useful for creating environments involving complex interaction between agents, while keeping the control and perception problems simple, as we are primarily interested in addressing agent interaction. To further simplify the control problem, we use discrete action spaces, allowing agents to move up, down, left, right, or stay; however, the agents may not immediately move exactly in the specified direction, as the task framework incorporates a basic physics engine where agents' momentums are taken into account. Fig. 2 illustrates the two environments. Cooperative Treasure Collection The cooperative environment in Figure 2a ) involves 8 total agents, 6 of which are "treasure hunters" and 2 of which are "treasure banks", which each correspond to a different color of treasure. The role of the hunters is to collect the treasure of any color, which re-spawn randomly upon being collected (with a total of 6), and then "deposit" the treasure into the correctly colored "bank". The role of each bank is to simply gather as much treasure as possible from the hunters. All agents are able to see each others' positions with respect to their own. Hunters receive a global reward for the successful collection of treasure and all agents receive a global reward for the depositing of treasure. Hunters are additionally penalized for colliding with each other. As such, the task contains a mixture of shared and individual rewards and requires different "modes of attention" which depend on the agent's state and other agents' potential for affecting its rewards. Rover-Tower The environment in Figure 2b involves 8 total agents, 4 of which are "rovers" and another 4 which are "towers". At each episode, rovers and towers are randomly paired. The pair is negatively rewarded by the distance of the rover to its goal. The task can be thought of as a navigation task on an alien planet with limited infrastructure and low visibility. The rovers are unable to see in their surroundings and must rely on communication from the towers, which are able to locate the rovers as well as their destinations and can send one of five discrete communication messages to their paired rover. Note that communication is highly restricted and different from centralized policy approaches BID14, which allow for free transfer of continuous information among policies. In our setup, the communication is integrated into the environment (in the tower's action BID17 space and the rover's observation space), rather than being explicitly part of the model, and is limited to a few discrete signals. We compare to two recently proposed approaches for centralized training of decentralized policies: MADDPG BID20 and COMA, as well as a single-agent RL approach, DDPG, trained separately for each agent. In order to enable learning in discrete action spaces for both MADDPG and DDPG, where deterministic policies are not possible, we use the Gumbel-Softmax reparametrization trick BID13. We will refer to these modified versions as MADDPG (Discrete) and DDPG (Discrete). For a detailed description of this reparametrization, see the appendix 6.2. We use soft actor critic to optimize. Thus, in order to have fair comparisons, we additionally implement MADDPG and COMA with Soft Actor-Critic, named as MADDPG+SAC and COMA+SAC.We also consider an ablated version of our model as a variant of our approach. In this model, we use uniform attention by fixing the attention weight α j (Eq. 6) to be 1/(N − 1). This restriction prevents the model from focusing its attention on specific agents. All methods are implemented such that their approximate total number of parameters (across agents) are equal to our method, and each model is trained with 6 random seeds each. Hyperparameters for each underlying algorithm are tuned based on performance and kept constant across all variants of critic architectures for that algorithm. A thorough comparison of all baselines is summarized in TAB0. FIG1 illustrates averaged rewards per episode by various methods. The proposed approach (MAAC) is competitive with other approaches being compared. In what follows, we provide detailed analysis. Uniform attention is competitive with our approach in the Cooperative Treasure Collection (CTC) environment, but not in Rover-Tower. On the other hand, both MADDPG (Discrete) and MADDPG+SAC perform well on Rover-Tower, though they do not on CTC. Both variants of COMA do not fare well in our environments. DDPG, arguably a weaker baseline, performs surprisingly well in CTC, but does poorly in Rover-Tower. In CTC, the rewards are shared across agents thus an agent's critic does not need to focus on information from specific agents in order to calculate its expected rewards. Moreover, each agent's local observation provides enough information to make a decent prediction of its expected rewards. This might explain why MAAC (uniform) which attends to other agents equally, and DDPG (being very unattentive to other agents) perform well. On the other hand, rewards in the Rover-Tower environment for a specific agent are tied to another single agent's observations. This environment exemplifies a class of scenarios where dynamic attention can be beneficial: when subgroups of agents are interacting and performing coordinated tasks with separate rewards, but the groups do not remain static. This explains why MAAC (uniform) perform poorly and DDPG completely breaks down, as knowing information from another specific agent is crucial in predicting expected rewards. COMA uses a single centralized network for predicting Q-values for all agents with separate forward passes. Thus, this approach may perform best in environments with global rewards and agents with similar action spaces. However, our environments have agents with differing roles (and non-global rewards in the case of Rover-Tower). Thus both variants of COMA do not fare well. MADDPG (and its variant) is a very strong method. However, we suspect its low performance in CTC is due to this environment's relatively large observation spaces for all agents, as the MADDPG critic concatenates observations for all agents into a single input vector for each agent's critic. Our next experiments confirm this hypothesis. Scalability We compare the average rewards attained by both approaches (normalized by the range of rewards attained in the environment, as differing the number of agents changes the nature of rewards in each environment), and show that the improvement of our approach MAAC over MADDPG+SAC grows with respect to the number of agents. As suspected, MADDPG-like critics use all information non-selectively, while our approach can learn which agents to pay more attention through the attention mechanism. Thus our approach scales better when the number of agents increases. In future research we will continue to improve the scalability when the number of agents further increases by sharing policies among agents, and performing attention on sub-groups (of agents).While the Rover-Tower task has a lot of agents, each agent only gets information about its paired agent -in other words, the task itself has an intrinsically smaller number of "other agents" (conditioned on each agent) than the CTC environment. As a future direction, we are creating more complicated environments where each agent needs to cope with a large group of agents where selective attention is needed. This naturally models real-life scenarios that multiple agents are organized in clusters/sub-societies (school, work, family, etc) where the agent needs to interact with a small number of agents from many groups. We anticipate that in such complicated scenarios, our approach, combined with some advantages exhibited by other approaches would do well. We propose an algorithm for training decentralized policies in multi-agent settings. The key idea is to utilize attention in order to select relevant information for estimating critics. We analyze the performance of the proposed approach with respect to the number of agents, different configurations of rewards, and the span of relevant observational information. Empirical are promising and we intend to extend to highly complicated and dynamic environments. for j = 1... num critic updates do Update target critic and policy parameters: DISPLAYFORM0 T update ← 0 Update critic: DISPLAYFORM1 2, where Update policies: DISPLAYFORM2 DISPLAYFORM3 38: end function We train using Soft Actor-Critic BID10, an off-policy, actor-critic method for maximum entropy reinforcement learning. Our training procedure consists of performing 12 parallel rollouts, and adding a tuple of (o t, a t, r t, o t+1) 1...N to a replay buffer (with maximum length 1e6) for each timepoint. We reset each environment after every 100 steps (an episode). After 100 steps (across all rollouts), we perform 4 updates for the attention critic and for all policies. For each update we sample minibatches of 1024 timepoints from the replay buffer and then perform gradient descent on the Q-function loss objective FORMULA11, as well as the policy objective, using Adam BID15 as the optimizer for both with a learning rate of 0.001. These updates can be computed efficiently in parallel (across agents) using a GPU. After the updates are complete, we update the parametersψ of our target critic Qψ to move toward our learned critic's parameters, ψ, as in Lillicrap et al. FORMULA2; BID10:ψ = (1 − τ)ψ + τ ψ, where τ is the update rate (set to 0.002 for attention parameters and 0.04 for all other parameters). Using a target critic has been shown to stabilize the use of experience replay for off-policy reinforcement learning with neural network function approximators BID17. We update the parameters of the target policies,θ in the same manner. We use a discount factor, γ, of 0.99. All networks (separate policies and contained within the centralized critics) use a hidden dimension of 128 and Leaky Rectified Linear Units as the nonlinearity. We use 0.2 as our temperature setting for Soft Actor-Critic. Additionally, we typically use 4 attention heads in our attention critics unless otherwise specified. In order to compare to DDPG and MADDPG in our environments with discrete action spaces, we must make a slight modification to the basic algorithm. This modification is first suggested by BID20 in order to enable policies that output discrete communication messages. Consider the original DDPG policy gradient which takes advantage of the fact that you can easily calculate the gradient of the output of a deterministic policy with respect to its parameters. DISPLAYFORM0 Rather than policies that deterministically output an action from within a continuous action space, we use policies that produce differentiable samples through a Gumbel-Softmax distribution BID13. Using differentiable samples allows us to use the gradient of expected returns to train policies without using the log derivative trick, just as in DDPG. a∼π(s) [∇ a Q(s, a)∇ θ a] DISPLAYFORM1 In order to understand how the use of attention evolves over the course of training, we examine the "entropy" of the attention weights for each agent for each of the four attention heads that we use in both tasks (Figures 4 and 5). The black bars indicate the maximum possible entropy (i.e. uniform attention across all agents). Lower entropy indicates that the head is focusing on specific agents, with an entropy of 0 indicating attention focusing on one agent. In Rover-Tower, we plot the attention entropy for each rover. Interestingly, each agent appears to use a different combination of the four heads, but their use is not mutually exclusive, indicating that the inclusion of separate attention heads for each agent is not necessary. This differential use of attention heads is sensible due to the nature of rewards in this environment (i.e. individualized rewards). In the case of Treasure Collection, we find that all agents use the attention heads similarly, which is unsurprising considering that rewards are shared in that environment. In order to inspect how the attention mechanism is working on a more fine-grained level, we visualize the attention weights for one of the rovers in Rover-Tower (Figure 6), from the head that the agent appears to use the most (determined by looking at Figure 4), while changing the tower that said rover is paired to. In these plots, we ignore the weights over other rovers for simplicity since these are always near zero. We find that the rover learns to strongly attend to the tower that it is paired with, without any explicit supervision signal to do so. The model implicitly learns which Figure 5: Attention "entropy" for each head over the course of training for two collectors in the Treasure Collection Environment agent is most relevant to estimating the rover's expecture future returns, and said agent can change dynamically without affecting the performance of the algorithm. In order to test our model's ability to handle continuous action spaces, we add a network for each agent to learn a state-value function V i (o, a \i), which uses the same weighted attention embedding over other agents as Q i (o, a). The loss functions to learn both networks are provided by BID10. We test on an environment introduced in BID20 called Cooperative Navigation and compare to MADDPG. Our are presented in TAB3. This task does not require attention, as all agents are relevant to each others rewards at each time step. As such, it is unsurprising that our Figure 6: Attention weights when subjected to different Tower pairings for Rover 1 in Rover-Tower environment approach matches but does not surpass the performance of MADDPG. It is notable, however, both that attention does not harm performance in simple cases and that our approach handles continuous action spaces as well.

represent the tldr for finding the corresponding scientific article

To give you a sense - "This paper proposes an effective method to compress neural networks based on recent results in information theory." should be close to "While deep neural networks are a highly successful model class, their large memory footprint puts considerable strain on energy consumption, communication bandwidth, and storage requirements. Consequently, model size reduction has become an utmost goal in deep learning. A typical approach is to train a set of deterministic weights, while applying certain techniques such as pruning and quantization, in order that the empirical weight distribution becomes amenable to Shannon-style coding schemes. However, as shown in this paper, relaxing weight determinism and using a full variational distribution over weights allows for more efficient coding schemes and consequently higher compression rates. In particular, following the classical bits-back argument, we encode the network weights using a random sample, requiring only a number of bits corresponding to the Kullback-Leibler divergence between the sampled variational distribution and the encoding distribution. By imposing a constraint on the Kullback-Leibler divergence, we are able to explicitly control the compression rate, while optimizing the expected loss on the training set. The employed encoding scheme can be shown to be close to the optimal information-theoretical lower bound, with respect to the employed variational family. Our method sets new state-of-the-art in neural network compression, as it strictly dominates previous approaches in a Pareto sense: On the benchmarks LeNet-5/MNIST and VGG-16/CIFAR-10, our approach yields the best test performance for a fixed memory budget, and vice versa, it achieves the highest compression rates for a fixed test performance. With the celebrated success of deep learning models and their ever increasing presence, it has become a key challenge to increase their efficiency. In particular, the rather substantial memory requirements in neural networks can often conflict with storage and communication constraints, especially in mobile applications. Moreover, as discussed in BID4, memory accesses are up to three orders of magnitude more costly than arithmetic operations in terms of energy consumption. Thus, compressing deep learning models has become a priority goal with a beneficial economic and ecological impact. Traditional approaches to model compression usually rely on three main techniques: pruning, quantization and coding. For example, Deep Compression BID5 proposes a pipeline employing all three of these techniques in a systematic manner. From an information-theoretic perspective, the central routine is coding, while pruning and quantization can be seen as helper heuristics to reduce the entropy of the empirical weight-distribution, leading to shorter encoding lengths BID15. Also, the recently proposed Bayesian Compression BID13 falls into this scheme, despite being motivated by the so-called bits-back argument BID8 which theoretically allows for higher compression rates.1 While the bits-back argument certainly motivated the use of variational inference in Bayesian Compression, the downstream encoding is still akin to Deep Compression (and other approaches). In particular, the variational distribution is merely used to derive a deterministic set of weights, which is subsequently encoded with Shannonstyle coding. This approach, however, does not fully exploit the coding efficiency postulated by the bits-back argument. In this paper, we step aside from the pruning-quantization pipeline and propose a novel coding method which approximately realizes bits-back efficiency. In particular, we refrain from constructing a deterministic weight-set but rather encode a random weight-set from the full variational posterior. This is fundamentally different from first drawing a weight-set and subsequently encoding it -this would be no more efficient than previous approaches. Rather, the coding scheme developed here is allowed to pick a random weight-set which can be cheaply encoded. By using from BID6, we show that such an coding scheme always exists and that the bits-back argument indeed represents a theoretical lower bound for its coding efficiency. Moreover, we propose a practical scheme which produces an approximate sample from the variational distribution and which can indeed be encoded with this efficiency. Since our algorithm learns a distribution over weightsets and derives a random message from it, while minimizing the ing code length, we dub it Minimal Random Code Learning (MIRACLE).From a practical perspective, MIRACLE has the advantage that it offers explicit control over the expected loss and the compression size. This is distinct from previous techniques, which require tedious tuning of various hyper-parameters and/or thresholds in order to achieve a certain coding goal. In our method, we can simply control the KL-divergence using a penalty factor, which directly reflects the achieved code length (plus a small overhead), while simultaneously optimizing the expected training loss. As a , we were able to trace the trade-off curve for compression size versus classification performance (FIG4). We clearly outperform previous state-of-the-art in a Pareto sense: For any desired compression rate, our encoding achieves better performance on the test set; vice versa, for a certain performance on the test set, our method achieves the highest compression. To summarize, our main contributions are:• We introduce MIRACLE, an innovative compression algorithm that exploits the noise resistance of deep learning models by training a variational distribution and efficiently encodes a random set of weights.• Our method is easy to implement and offers explicit control over the loss and the compression size.• We provide theoretical justification that our algorithm gets close to the theoretical lower bound on the encoding length.• The potency of MIRACLE is demonstrated on two common compression tasks, where it clearly outperforms previous state-of-the-art methods for compressing neural networks. In the following section, we discuss related work and introduce required . In Section 3 we introduce our method. Section 4 presents our experimental and Section 5 concludes the paper. There is an ample amount of research on compressing neural networks, so that we will only discuss the most prominent ones, and those which are related to our work. An early approach is Optimal Brain Damage which employs the Hessian of the network weights in order to determine whether weights can be pruned without significantly impacting training performance. A related but simpler approach was proposed in BID4, where small weights are truncated to zero, alternated with re-training. This simple approach yielded -somewhat surprisingly -networks which are one order of magnitude smaller, without impairing performance. The approach was refined into a systematic pipeline called Deep Compression, where magnitude-based weight pruning is followed by weight quantization (clustering weights) and Huffman coding BID10. While a variational posterior. However, in order to realize this effective cost, one needs to encode both the network weights and the training targets, while it remains unclear whether it can also be achieved for network weights alone.its compression ratio (∼ 50×) has been surpassed since, many of the subsequent works took lessons from this paper. HashNet proposed by BID1 also follows a simple and surprisingly effective approach: They exploit the fact that training of neural networks is resistant to imposing random constraints on the weights. In particular, they use hashing to enforce groups of weights to share the same value, yielding memory reductions of up to 64× with gracefully degrading performance. Weightless encoding by BID14 demonstrates that neural networks are resilient to weight noise, and exploits this fact for a lossy compression algorithm. The recently proposed Bayesian Compression BID13 ) uses a Bayesian variational framework and is motivated by the bits-back argument BID8. Since this work is the closest to ours, albeit with important differences, we discuss Bayesian Compression and the bits-back argument in more detail. The basic approach is to equip the network weights w with a prior p and to approximate the posterior using the standard variational framework, i.e. maximize the evidence lower bound (ELBO) for a given dataset DISPLAYFORM0 w.r.t. the variational distribution q φ, parameterized by φ. The bits-back argument BID8 ) establishes a connection between the Bayesian variational framework and the Minimum Description Length (MDL) principle BID3. Assuming a large dataset D of input-target pairs, we aim to use the neural network to transmit the targets with a minimal message, while the inputs are assumed to be public. To this end, we draw a weight-set w * from q φ, which has been obtained by maximizing FORMULA0; note that knowing a particular weight w * set conveys a message of length H[q φ] (H refers to the Shannon entropy of the distribution). The weight-set w * is used to encode the residual of the targets, and is itself encoded with the prior distribution p, yielding a message of length DISPLAYFORM1. This message allows the receiver to perfectly reconstruct the original targets, and consequently the variational distribution q φ, by running the same (deterministic) algorithm as used by the sender. Consequently, with q φ at hand, the receiver is able to retrieve an auxiliary message encoded in w *. When subtracting the length of this "free message" from the original E q φ [log p] nats, 2 we yield a net cost of KL(q φ ||p) = E q φ [log q φ p] nats for encoding the weights, i.e. we recover the ELBO as negative MDL BID8.In BID9 BID2 coding schemes were proposed which practically exploited the bits-back argument for the purpose of coding data. However, it is not clear how these free bits can be spent solely for the purpose of model compression, as we only want to store a representation of our model, while discarding the training data. Therefore, while Bayesian Compression is certainly motivated by the bits-back argument, it actually does not strive for the postulated coding efficiency KL(q φ ||p). Rather, this method imposes a sparsity inducing prior distribution to aid the pruning process. Moreover, high posterior variance is translated into reduced precision which constitutes a heuristic for quantization. In the end, Bayesian Compression merely produces a deterministic weight-set w * which is encoded similar as in preceding works. In particular, all previous approaches essentially use the following coding scheme, or a (sometimes sub-optimal) variant of it. After a deterministic weight-set w * has been obtained, involving potential pruning and quantization techniques, one interprets w * as a sequence of i.i.d. variables, taking values from a finite alphabet. Then one assumes the coding distribution DISPLAYFORM2 where δ x denotes the Kronecker delta at x. According to Shannon's source coding theorem BID15, w * can be coded with no less than N H[p] nats, which is asymptotically achieved by Huffman coding, like in BID5. Note that the Shannon lower bound can be written as DISPLAYFORM3 where we have set p (w) = i p (w i). Thus, these Shannon-style coding schemes are in some sense optimal, when the variational family is restricted to point-measures, i.e. deterministic weights. By extending the variational family to comprise more general distributions q, the coding length KL(q||p) could be drastically reduced. In the following, we develop such a method which exploits the uncertainty represented by q in order to encode a random weight-set with short coding length. Consider the scenario where we want to train a neural network but our memory budget is constrained to C nats. As illustrated in the previous section, a variational approach offers -in principle -a simple and elegant solution. Before we proceed, we note that we do not consider our approach to be a strictly Bayesian one, but rather based on the MDL principle, although these two are of course highly related BID3. In particular, we refer to p as an encoding distribution rather than a prior, and moreover we will use a framework akin to the β-VAE BID7 which better reflects our goal of efficient coding. The crucial difference to the β-VAE being that we encode parameters rather than data. Now, similar to BID13, we first fix a suitable network architecture, select an encoding distribution p and a parameterized variational family q φ for the network weights w. We consider, however, a slightly different variational objective related to the β-VAE: DISPLAYFORM0 This objective directly reflects our goal of achieving both a good training performance (loss term) and being able to represent our model with a short code (model complexity), at least according to the bits-back argument. After obtaining q φ by maximizing, a weight-set drawn from q φ will perform comparable to a deterministically trained network, since the variance of the negative loss term will be comparatively small to the mean, and since the KL term regularizes the model. Thus, our declared goal is to draw a sample from q φ such that this sample can be encoded as efficiently as possible. This problem can be formulated as the following communication problem. Alice observes a training data set (X, Y) = D drawn from an unknown distribution p(D). She trains a variational distribution q φ (w) by optimizing for a given β using a deterministic algorithm. Subsequently, she wishes to send a message M (D) to Bob, which allows him to generate a sample distributed according to q φ. How long does this message need to be?The answer to this question depends on the unknown data distribution p(D), so we need to make an assumption about it. Since the variational parameters φ depend on the realized dataset D, we can interpret the variational distribution as a conditional distribution q(w|D):= q φ (w), giving rise to the joint q(w, D) = q(w|D)p(D). Now, our assumption about p(D) is that q(w|D)p(D) dD = p(w), that is, the variational distribution q φ yields the assumed encoding distribution p(w), when averaged over all possible datasets. Note that this a similar strong assumption as in a Bayesian setting, where we assume that the data distribution is given as p(D) = p(D|w)p(w)dw. In this setting, it follows immediately from the data processing inequality BID6 ) that in expectation the message length |M | cannot be smaller than KL(q φ ||p): DISPLAYFORM1 where I refers to the mutual information and in the third inequality we applied the data processing inequality for Markov chain D → M → w. As discussed by BID6, the inequal- DISPLAYFORM2 can be very loose. However, as they further show, the message length can be brought close to the lower bound, if Alice and Bob are allowed to share a source of randomness:Theorem 3.1 BID6 ) Given random variables D, w and a random string R, let a protocol Π be defined via a message function M (D, R) and a decoder function w(M, R), DISPLAYFORM3 be the expected message length for data D, and let the minimal expected message length be defined as DISPLAYFORM4 where Π The of BID6 establish a characterization of the mutual information in terms of minimal coding a conditional sample. For our purposes, Theorem 3.1 guarantees that in principle DISPLAYFORM5 draw a sample w k * ∼q 7:return w k *, k * 8: end procedure there is an algorithm which realizes near bits-back efficiency. Furthermore, the theorem shows that this is indeed a fundamental lower bound, i.e. that such an algorithm is optimal for the considered setting. To this end, we need to refer to a "common ground", i.e. a shared random source R, where w.l.o.g. we can assume that this source is an infinite list of samples from our encoding distribution p. In practice, this can be realized via a pseudo-random generator with a public seed. While BID6 provide a constructive proof using a variant of rejection sampling (see Appendix A), this algorithm is in fact intractable, because it requires keeping track of the acceptance probabilities over the whole sample domain. Therefore, we propose an alternative method to produce an approximate sample from q φ, depicted in Algorithm 1. This algorithm takes as inputs the trained variational distribution q φ and the encoding distribution p. We first draw K = exp(KL(q φ ||p)) samples from p, using the shared random generator. Subsequently, we craft a discrete proxy distributioñ q, which has support only on these K samples, and where the probability mass for each sample is proportional to the importance weights a k = q φ (w k) p(w k). Finally, we draw a sample fromq and return its index k * and the sample w k * itself. Since any number 0 ≤ k * < K can be easily encoded with KL(q φ ||p) nats, we achieve our aimed coding efficiency. Decoding the sample is easy: simply draw the k * th sample w k * from the shared random generator (e.g. by resetting the random seed).While this algorithm is remarkably simple and easy to implement, there is of course the question of whether it is a correct thing to do. Moreover, an immediate caveat is that the number K of required samples grows exponentially in KL(q φ ||p), which is clearly infeasible for encoding a practical neural network. The first point is addressed in the next section, while the latter is discussed in Section 3.3, together with other practical considerations. The proxy distributionq in Algorithm 1 is based on an importance sampling scheme, as its probability masses are defined to be proportional to the usual importance weights DISPLAYFORM0. Under mild assumptions (q φ, p continuous; a k < ∞) it is easy to verify thatq converges to q φ in distribution for K → ∞; thus in the limit, Algorithm 1 samples from the correct distribution. However, since we collect only K = exp(KL(q φ ||p)) samples in order to achieve a short coding length,q will be biased. Fortunately, it turns out that K is just in the right order for this bias to be small. DISPLAYFORM1. Furthermore, let f (w) be a measurable function and ||f || q φ = E q φ [f 2] be its 2-norm under q φ. Then it holds that DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 Perform stochastic gradient update of L O 19: DISPLAYFORM5 else 23: DISPLAYFORM6 DISPLAYFORM7 which is precisely the importance sampling estimator for unnormalized distributions (denoted as J n in BID0), i.e. their Theorem 1.2 directly yields Theorem 3.2. Note that the term e − t /4 decays quickly with t, and, since log q φ /p is typically concentrated around its expected value KL(q||p), the second term in also quickly becomes negligible. Thus, roughly speaking, Theorem 3.2 establishes that E q φ [f] ≈ Eq[f] with high probability, for any measurable function f. This is in particular true for the function f (w) = log p(D|w) − β log q φ (w) p(w). Note that the expectation of this function is just the variational objective we optimized to yield q φ in the first place. Thus, DISPLAYFORM8, replacing q φ byq is well justified. Thereby, any sample ofq can trivially be encoded with KL(q φ ||p) nats, and decoded by simple reference to a pseudo-random generator. Note that according to Theorem 3.2 we should actually take a number of samples somewhat larger than exp(KL(q φ ||p)) in order to make sufficiently small. In particular, the in BID0 ) also imply that a too small number of samples will typically be quite off the targeted expectation (for the worst-case f). However, although our choice of number of samples is at a critical point, in our experiments this number of samples yielded very good . In this section, we describe the application of Algorithm 1 within a practical learning algorithmMinimal Random Code Learning (MIRACLE) -depicted in Algorithm 2. For both q φ and p we used Gaussians with diagonal covariance matrices. For q φ, all means and standard deviations constituted the variational parameters φ. The mean of p was fixed to zero, and the standard deviation was shared within each layer of the encoded network. These shared parameters of p where learned jointly with q φ, i.e. the encoding distribution was also adapted to the task. This choice of distributions allowed us to use the reparameterization trick for effective variational training and furthermore, KL(q φ ||p) can be computed analytically. Since generating K = exp(KL(q φ ||p)) samples is infeasible for any reasonable KL(q φ ||p), we divided the overall problem into sub-problems. To this end, we set a global coding goal of C nats and a local coding goal of C loc nats. We randomly split the weight vector w into B = C C loc equally sized blocks, and assigned each block an allowance of C loc nats. For example, fixing C loc to 11.09 nats ≈ 16 bits, corresponds to K = 65536 samples which need to be drawn per block. We imposed block-wise KL constraints using block-wise penalty factors β b, which were automatically annealed via multiplication/division with (1 + β) during the variational updates (see Algorithm 2). Note that the random splitting into B blocks can be efficiently coded via the shared random generator, and only the number B needs communicated. Before encoding any weights, we made sure that variational learning had converged by training for a large number of iterations I 0 = 10 4. After that, we alternated between encoding single blocks and updating the variational distribution not-yet coded weights, by spending I intermediate variational iterations. To this end, we define a variational objective L O w.r.t. to blocks which have not been coded yet, while weights of already encoded blocks were fixed to their encoded value. Intuitively, this allows to compensate for poor choices in earlier encoded blocks, and was crucial for good performance. Theoretically, this amounts to a rich auto-regressive variational family q φ, as the blocks which remain to be updated are effectively conditioned on the weights which have already been encoded. We also found that the hashing trick BID1 further improves performance (not depicted in Algorithm 2 for simplicity). The hashing trick randomly conditions weights to share the same value. While BID1 apply it to reduce the entropy, in our case it helps to restrict the optimization space and reduces the dimensionality of both p and q φ. We found that this typically improves the compression rate by a factor of ∼ 1.5×. The experiments 3 were conducted on two common benchmarks: LeNet-5 on MNIST and VGG-16 on CIFAR-10. As baselines we used three recent state-of-the-art methods, namely Deep Compression BID5, Weightless encoding BID14 and Bayesian Compression BID13. The performance of the baseline methods are quoted from their respective source materials. BID11 with the default learning rate (10 −3) and we set β0 = 10 −8 and β = 5 × 10 −5. For VGG, the means of the weights were initialized using a pretrained model. 4 We recommend applying the hashing trick mainly to reduce the size of the largest layers. In particular, we applied the hashing trick was to layers 2 and 3 in LeNet-5 to reduce their sizes by 2× and 64× respectively and to layers 10-16 in VGG to reduce their sizes 8×. The local coding goal C loc was fixed at 20 bits for LeNet-5 and it was varied between 15 and 5 bits for VGG (B was kept constant). For the number of intermediate variational updates I, we used I = 50 for LeNet-5 and I = 1 for VGG, in order to keep training time reasonable (≈ 1 day on a single NVIDIA P100 for VGG).The performance trade-offs (test error rate and compression size) of MIRACLE along with the baseline methods and the uncompressed model are shown in FIG4 and TAB1. For MIRACLE we can easily construct the Pareto frontier, by starting with a large coding goal C (i.e. allowing a large coding length) and successively reducing it. Constructing such a Pareto frontier for other methods is delicate, as it requires re-tuning hyper-parameters which are often only indirectly related to the compression size -for MIRACLE it is directly reflected via the KL-term. We see that MIRACLE is Pareto-better than the competitors: for a given test error rate, we achieve better compression, while for a given model size we achieve lower test error. In this paper we followed through the philosophy of the bits-back argument for the goal of coding model parameters. The basic insight here is that restricting to a single deterministic weight-set and aiming to coding it in a classic Shannon-style is greedy and in fact sub-optimal. Neural networks -and other deep learning models -are highly overparameterized, and consequently there are many "good" parameterizations. Thus, rather than focusing on a single weight set, we showed that this fact can be exploited for coding, by selecting a "cheap" weight set out of the set of "good" ones. Our algorithm is backed by solid recent information-theoretic insights, yet it is simple to implement. We demonstrated that the presented coding algorithm clearly outperforms previous state-of-the-art. An important question remaining for future work is how efficient MIRACLE can be made in terms of memory accesses and consequently for energy consumption and inference time. There lies clear potential in this direction, as any single weight can be recovered by its block-index and relative index within each block. By smartly keeping track of these addresses, and using pseudo-random generators as algorithmic lookup-tables, we could design an inference machine which is able to directly run our compressed models, which might lead to considerable savings in memory accesses. This is shown by proving that q(w) − p i (w) ≤ q(w)(1 − p(w)) i for i ∈ N.In order to bound the encoding length, one has to first show that if the accepted sample has index i *, then E[log i *] ≤ KL(q||p) + O.Following this, one can employ the prefix-free binary encoding of BID16. Let l(n) be the length of the encoding for n ∈ N using the encoding scheme proposed by BID16. Their method is proven to have |l(n)| = log n + 2 log log(n + 1) + O, from which the upper bound follows: DISPLAYFORM0"

We propose an approach to learn decentralized policies in multi-agent settings using attention-based critics and demonstrate promising results in environments with complex interactions.

represent the short summary for finding the corresponding scientific article (long)!
------
For example, Empirical and theoretical study of the effects of staleness in non-synchronous execution on machine learning algorithms. should be similar to Most distributed machine learning (ML) systems store a copy of the model parameters locally on each machine to minimize network communication. In practice, in order to reduce synchronization waiting time, these copies of the model are not necessarily updated in lock-step, and can become stale. Despite much development in large-scale ML, the effect of staleness on the learning efficiency is inconclusive, mainly because it is challenging to control or monitor the staleness in complex distributed environments. In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates. Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature. The empirical findings also inspire a new convergence analysis of SGD in non-convex optimization under staleness, matching the best-known convergence rate of O(1/\sqrt{T}). With the advent of big data and complex models, there is a growing body of works on scaling machine learning under synchronous and non-synchronous 1 distributed execution BID8 BID11 BID29. These works, however, point to seemingly contradictory on whether non-synchronous execution outperforms synchronous counterparts in terms of absolute convergence, which is measured by the wall clock time to reach the desired model quality. For deep neural networks, BID2; BID8 show that fully asynchronous systems achieve high scalability and model quality, but others argue that synchronous training converges faster BID1 BID5. The disagreement goes beyond deep learning models:; BID49; BID26; BID31; BID41 empirically and theoretically show that many algorithms scale effectively under non-synchronous settings, but BID36;; demonstrate significant penalties from asynchrony. The crux of the disagreement lies in the trade-off between two factors contributing to the absolute convergence: statistical efficiency and system throughput. Statistical efficiency measures convergence per algorithmic step (e.g., a mini-batch), while system throughput captures the performance of the underlying implementation and hardware. Non-synchronous execution can improve system throughput due to lower synchronization overheads, which is well understood BID1 BID4 BID2. However, by allowing various workers to use stale versions of the model that do not always reflect the latest updates, non-synchronous systems can exhibit lower statistical efficiency BID1 BID5. How statistical efficiency and system throughput trade off in distributed systems, however, is far from clear. The difficulties in understanding the trade-off arise because statistical efficiency and system throughput are coupled during execution in distributed environments. Non-synchronous executions are in general non-deterministic, which can be difficult to profile. Furthermore, large-scale experiments 2 RELATED WORK Staleness is reported to help absolute convergence for distributed deep learning in BID2; BID8; and has minimal impact on convergence BID31 BID6 BID51 BID32. But BID1; BID5 show significant negative effects of staleness. LDA training is generally insensitive to staleness BID44 BID47 BID7, and so is MF training BID48 BID33 BID4 BID49. However, none of their evaluations quantifies the level of staleness in the systems. By explicitly controlling the staleness, we decouple the distributed execution, which is hard to control, from ML convergence outcomes. We focus on algorithms that are commonly used in large-scale optimization BID11 BID1 BID8, instead of methods specifically designed to minimize synchronization BID39 BID43 BID20. Non-synchronous execution has theoretical underpinning BID30 BID49 BID31 BID41. Here we study algorithms that do not necessarily satisfy assumptions in their analyses. We study six ML models and focus on algorithms that lend itself to data parallelism, which a primary approach for distributed ML. Our algorithms span optimization, sampling, and black box variational inference. TAB1 summarizes the studied models and algorithms. Simulation Model. Each update generated by worker p needs to be propagated to both worker p's model cache and other worker's model cache. We apply a uniformly random delay model to these updates that are in transit. Specifically, let u each worker p (including p itself), our delay model applies a delay r BID40 BID9 BID23 BID16 BID12, and dataset BID24 BID34 BID27 BID14 BID42 in our study. η denotes learning rate, which, if not specified, are tuned empirically for each algorithm and staleness level, β1, β2 are optimization hyperparameters (using common default values). α, β in LDA are Dirichlet priors for document topic and word topic random variables, respectively. Convolutional Neural Networks (CNNs) have been a strong focus of large-scale training, both under synchronous BID11 BID5 BID3 and non-synchronous BID2 BID8 BID1 training. We consider residual networks with 6n + 2 weight layers BID15. The networks consist of 3 groups of n residual blocks, with 16, 32, and 64 feature maps in each group, respectively, followed by a global pooling layer and a softmax layer. The residual blocks have the same construction as in BID15. We measure the model quality using test accuracy. For simplicity, we omit data augmentation in our experiments. Deep Neural Networks (DNNs) are neural networks composed of fully connected layers. Our DNNs have 1 to 6 hidden layers, with 256 neurons in each layer, followed by a softmax layer. We use rectified linear units (ReLU) for nonlinearity after each hidden layer BID38. Multiclass Logistic Regression (MLR) is the special case of DNN with 0 hidden layers. We measure the model quality using test accuracy. Matrix factorization (MF) is commonly used in recommender systems and have been implemented at scale BID48 BID33 BID4 BID49 BID22. Let D ∈ R M ×N be a partially filled matrix, MF factorizes D into two factor matrices L ∈ R M ×r and R ∈ R N ×r (r min(M, N) is the user-defined rank). The 2 -penalized optimization problem is: DISPLAYFORM0 where || · || F is the Frobenius norm and λ is the regularization parameter. We partition observations D to workers while treating L, R as shared model parameters. We optimize MF via SGD, and measure model quality by training loss defined by the objective function above. Latent Dirichlet Allocation (LDA) is an unsupervised method to uncover hidden semantics ("topics") from a group of documents, each represented as a bag of tokens. LDA has been scaled under non-synchronous execution BID0 BID33 BID47 with great success. Further details are provided in Appendix. Variational Autoencoder (VAE) is commonly optimized by black box variational inference, which can be considered as a hybrid of optimization and sampling methods. The inputs to VAE training include two sources of stochasticity: the data sampling x and samples of random variable. We measure the model quality by test loss. We use DNNs with 1∼3 layers as the encoders and decoders in VAE, in which each layer has 256 units furnished with rectified linear function for non-linearity. The model quality is measured by the training objective value, assuming continuous input x and isotropic Gaussian prior p(z) ∼ N (0, I). We use batch size 32 for CNNs, DNNs, MLR, and VAEs 34. For MF, we use batch size of 25000 samples, which is 2.5% of the MovieLens dataset (1M samples). We study staleness up to s = 50 on 8 workers, which means model caches can miss updates up to 8.75 data passes. For LDA we use DISPLAYFORM0 as the batch size, where D is the number of documents and P is the number of workers. We study staleness up to s = 20, which means model caches can miss updates up to 2 data passes. We measure time in terms of the amount of work performed, such as the number of batches processed. Convergence Slowdown. Perhaps the most prominent effect of staleness on ML algorithms is the slowdown in convergence, evident throughout the experiments. FIG1 shows the number of batches needed to reach the desired model quality for CNNs and DNNs/MLR with varying network depths and different staleness (s = 0, ..., 16). FIG1 (d) show that convergence under higher level of staleness requires more batches to be processed in order to reach the same model quality. This additional work can potentially be quite substantial, such as in FIG1 where it takes up to 6x more batches compared with settings without staleness (s = 0). It is also worth pointing out that while there can be a substantial slowdown in convergence, the optimization still reaches desirable models under most cases in our experiments. When staleness is geometrically distributed FIG3 ), we observe similar patterns of convergence slowdown. We are not aware of any prior work reporting slowdown as high as observed here. This finding has important ramifications for distributed ML. Usually, the moderate amount of workload increases due to parallelization errors can be compensated by the additional computation resources and higher system throughput in the distributed execution. However, it may be difficult to justify spending large 3 Non-synchronous execution allows us to use small batch sizes, eschewing the potential generalization problem with large batch SGD BID21 BID35. 4 We present RNN in the Appendix.amount of resources for a distributed implementation if the statistical penalty is too high, which should be avoided (e.g., by staleness minimization system designs or synchronous execution).Model Complexity. FIG1 also reveals that the impact of staleness can depend on ML parameters, such as the depths of the networks. Overall we observe that staleness impacts deeper networks more than shallower ones. This holds true for SGD, Adam, Momentum, RMSProp, Adagrad FIG1, and other optimization schemes, and generalizes to other numbers of workers (see Appendix) 5.This is perhaps not surprising, given the fact that deeper models pose more optimization challenges even under the sequential settings BID10 BID15, though we point out that existing literature does not explicitly consider model complexity as a factor in distributed ML BID31 BID11. Our suggest that the staleness level acceptable in distributed training can depend strongly on the complexity of the model. For sufficiently complex models it may be more advantageous to eliminate staleness altogether and use synchronous training. Algorithms' Sensitivity to Staleness. Staleness has uneven impacts on different SGD variants. Fig. 2 shows the amount of work (measured in the number of batches) to reach the desired model quality for five SGD variants. Fig. 2 (d)(e)(f) reveals that while staleness generally increases the number of batches needed to reach the target test accuracy, the increase can be higher for certain algorithms, such as Momentum. On the other hand, Adagrad appear to be robust to staleness 6. Our finding is consistent with the fact that, to our knowledge, all existing successful cases applying non-synchronous training to deep neural networks use SGD BID8 BID2. In contrast, works reporting subpar performance from non-synchronous training often use momentum, such as RMSProp with momentum BID1 and momentum BID5. Our suggest that these different outcomes may be partly driven by the choice of optimization algorithms, leading to the seemingly contradictory reports of whether non-synchronous execution is advantageous over synchronous ones. Effects of More Workers. The impact of staleness is amplified by the number of workers. In the case of MF, Fig. 3(b) shows that the convergence slowdown in terms of the number of batches (normalized by the convergence for s = 0) on 8 workers is more than twice of the slowdown on 4 workers. For example, in Fig. 3 (b) the slowdown at s = 15 is ∼3.4, but the slowdown at the same staleness level on 8 workers is ∼8.2. Similar observations can be made for CNNs (Fig. 3). This can be explained by the fact that additional workers amplifies the effect of staleness by generating updates that will be subject to delays, and missing updates from other workers that are subject to delays. Fig. 3 (c)(d) show the convergence curves of LDA with different staleness levels for two settings varying on the number of workers and topics. Unlike the convergence curves for SGD-based algorithms (see Appendix), the convergence curves of Gibbs sampling are highly smooth, even under high staleness and a large number of workers. This can be attributed to the structure of log likelihood objective function BID12. Since in each sampling step we only update the count statistics based on a portion of the corpus, the objective value will generally change smoothly. Staleness levels under a certain threshold (s ≤ 10) lead to convergence, following indistinguishable log likelihood trajectories, regardless of the number of topics (K = 10, 100) or the number of workers (2-16 workers, see Appendix). Also, there is very minimal variance in those trajectories. However, for staleness beyond a certain level (s ≥ 15), Gibbs sampling does not converge to a fixed point. The convergence trajectories are distinct and are sensitive to the number of topics and the number of workers. There appears to be a "phase transition" at a certain staleness level that creates two distinct phases of convergence behaviors 7. We believe this is the first report of a staleness-induced failure case for LDA Gibbs sampling. Fig. 3 (e)(f), VAEs exhibit a much higher sensitivity to staleness compared with DNNs (FIG1). This is the case even considering that VAE with depth 3 has 6 weight layers, which has a comparable number of model parameters and network architecture to DNNs with 6 layers. We hypothesize that this is caused by the additional source of stochasticity from the sampling procedure, in addition to the data sampling process. We now provide theoretical insight into the effect of staleness on the observed convergence slowdown. We focus on the challenging asynchronous SGD (Async-SGD) case, which characterizes the neural network models, among others. Consider the following nonconvex optimization problem DISPLAYFORM0 where f i corresponds to the loss on the i-th data sample, and the objective function is assumed to satisfy the following standard conditions:Assumption 1. The objective function F in the problem (P) satisfies:1. Function F is continuously differentiable and bounded below, i.e., inf x∈R d F (x) > −∞; 2. The gradient of F is L-Lipschitz continuous. Notice that we allow F to be nonconvex. We apply the Async-SGD to solve the problem (P). Let ξ(k) be the mini-batch of data indices sampled from {1, . . ., n} uniformly at random by the algorithm at iteration k, and |ξ(k)| is the mini-batch size. Denote mini-batch gradient as ∇f ξ(k) (x k):= i∈ξ(k) ∇f i (x k). Then, the update rule of Async-SGD can be written as DISPLAYFORM1 where η k corresponds to the stepsize, τ k denotes the delayed clock and the maximum staleness is assumed to be bounded by s. This implies that DISPLAYFORM2 The optimization dynamics of Async-SGD is complex due to the nonconvexity and the uncertainty of the delayed updates. Interestingly, we find that the following notion of gradient coherence provides insights toward understanding the convergence property of Async-SGD.Definition 1 (Gradient coherence). The gradient coherence at iteration k is defined as DISPLAYFORM3 Parameter µ k captures the minimum coherence between the current gradient ∇F (x k) and the gradients along the past s iterations 8. Intuitively, if µ k is positive, then the direction of the current gradient is well aligned to those of the past gradients. In this case, the convergence property induced by using delayed stochastic gradients is close to that induced by using synchronous stochastic gradients. Note that Definition 1 only requires the gradients to be positively correlated over a small number of iterations s, which is often very small (e.g. <10 in our experiments). Therefore, Definition 1 is not a global requirement on optimization path. Even though neural network's loss function is non-convex, recent studies showed strong evidences that SGD in practical neural network training encourage positive gradient coherence BID28 BID32. This is consistent with the findings that the loss surface of shallow networks and deep networks with skip connections are dominated by large, flat, nearly convex attractors around the critical points BID28 BID21, implying that the degree of non-convexity is mild around critical points. We show in the sequel that µ k > 0 through most of the optimization path, especially when the staleness is minimized in practice by system optimization FIG3. Our theory can be readily adapted to account for a limited amount of negative µ k (see Appendix), but our primary interest is to provide a quantity that is easy to compute empirically during the course of optimization 9, and informative for the impact of staleness and can potentially be used to control synchronization levels. We now characterize the convergence property of Async-SGD. Theorem 1. Let Assumption 1 hold. Suppose for some µ > 0, the gradient coherence satisfies µ k ≥ µ for all k and the variance of the stochastic gradients is bounded by σ 2 > 0. Choose stepsize DISPLAYFORM4. Then, the iterates generated by the Async-SGD satisfy DISPLAYFORM5 8 Our gradient coherence bears similarity with the sufficient direction assumption in BID19. However, sufficient direction is a layer-wise and fixed delay, whereas our staleness is a random variable that is subject to system level factors such as communication bandwidth 9 It can be approximated by storing a pre-selected batch of data on a worker. The worker just needs to compute gradient every T mini-batches to obtain approximate ∇F (x k), ∇F (xt) in Definition 1. TAB1. Shaded region is 1 standard deviation over 3 runs. For computational efficiency, we approximate the full gradient ∇F (x k) by gradients on a fixed set of 1000 training samples D f ixed and use ∇D f ixed F (x k). (c) The number of batches to reach 71% test accuracy on CIFAR10 for ResNet8-32 using 8 workers and SGD under geometric delay distribution (details in Appendix).We refer readers to Appendix for the the proof. Theorem 1 characterizes several theoretical aspects of Async-SGD. First, the choice of the stepsize η k = µ sL √ k is adapted to both the maximum staleness and the gradient coherence. Intuitively, if the system encounters a larger staleness, then a smaller stepsize should be used to compensate the negative effect. On the other hand, the stepsize can be accordingly enlarged if the gradient coherence along the iterates turns out to be high. In this case, the direction of the gradient barely changes along the past several iterations, and a more aggressive stepsize can be adopted. In summary, the choice of stepsize should trade-off between the effects caused by both the staleness and the gradient coherence. depths optimized by SGD using 8 workers. The x-axis m is defined in FIG3 Furthermore, Theorem 1 shows that the minimum gradient norm decays at the rate O(DISPLAYFORM6), implying that the Async-SGD converges to a stationary point provided a positive gradient coherence, which we observe empirically in the sequel. On the other hand, the bound in Eq. captures the trade-off between the maximum staleness s and the gradient coherence µ. Specifically, minimizing the right hand side of Eq. with regard to the maximum staleness s yields the optimal choice s * = σµ log T L(F (x0)−infx F (x)), i.e., a larger staleness is allowed if the gradients remain to be highly coherent along the past iterates. Empirical Observations. Theorem 1 suggests that more coherent gradients along the optimization paths can be advantageous under non-synchronous execution. FIG3 shows the cosine similarity sim(a, b):= a·b a b between gradients along the convergence path for CNNs and DNNs 10. We observe the followings: Cosine similarity improves over the course of convergence FIG3 ). Except the highest staleness during the early phase of convergence, cosine similarity remains positive 11. In practice the staleness experienced during run time can be limited to small staleness, which minimizes the likelihood of negative gradient coherence during the early phase. FIG4 shows that cosine similarity decreases with increasing CNN model complexity. Theorem 1 implies that lower gradient coherence amplifies the effect of staleness s through the factor s µ 2 in Eq.. This is consistent with the convergence difficulty encountered in deeper models FIG1. In this work, we study the convergence behaviors under delayed updates for a wide array of models and algorithms. Our extensive experiments reveal that staleness appears to be a key governing parameter in learning. Overall staleness slows down the convergence, and under high staleness levels the convergence can progress very slowly or fail. The effects of staleness are highly problem 10 Cosine similarity is closely related to the coherence measure in Definition 1. 11 Low gradient coherence during the early part of optimization is consistent with the common heuristics to use fewer workers at the beginning in asynchronous training. BID31 also requires the number of workers to follow DISPLAYFORM0 where K is the iteration number.dependent, influenced by model complexity, choice of the algorithms, the number of workers, and the model itself, among others. Our empirical findings inspire new analyses of non-convex optimization under asynchrony based on gradient coherence, matching the existing rate of O(1/ √ T).Our findings have clear implications for distributed ML. To achieve actual speed-up in absolute convergence, any distributed ML system needs to overcome the slowdown from staleness, and carefully trade off between system throughput gains and statistical penalties. Many ML methods indeed demonstrate certain robustness against low staleness, which should offer opportunities for system optimization. Our support the broader observation that existing successful nonsynchronous systems generally keep staleness low and use algorithms efficient under staleness. A.1 PROOF OF THEOREM 1Theorem 2. Let Assumption 1 hold. Suppose the gradient coherence µ k is lower bounded by some µ > 0 for all k and the variance of the stochastic gradients is upper bounded by some σ 2 > 0. DISPLAYFORM0. Then, the iterates generated by the Async-SGD satisfy DISPLAYFORM1 Proof. By the L-Lipschitz property of ∇F, we obtain that for all k DISPLAYFORM2 Taking expectation on both sides of the above inequality and note that the variance of the stochastic gradient is bounded by σ 2, we further obtain that DISPLAYFORM3 Telescoping the above inequality over k from 0 to T yields that DISPLAYFORM4 Rearranging the above inequality and note that DISPLAYFORM5 Note that the choice of stepsize guarantees that η k µ − > 0 for all k. Thus, we conclude that DISPLAYFORM6 where the last inequality uses the fact that DISPLAYFORM7 into the above inequality and simplifying, we finally obtain that DISPLAYFORM8 A.2 HANDLING NEGATIVE GRADIENT COHERENCE IN THEOREM 1Our assumption of positive gradient coherence (GC) is motivated by strong empirical evidence that GC is largely positive FIG3 in the main text). Contrary to conventional wisdom, GC generally improves when approaching convergence for both SGD and Adam. Furthermore, in practice, the effective staleness for any given iteration generally concentrates in low staleness for the non-stragglers.When some µ k are negative at some iterations, in eq. 11 in the Appendix we can move the negative terms in k η k µ k to the right hand side and yield a higher upper bound (i.e., slower convergence). This is also consistent with empirical observations that higher staleness lowers GC and slows convergence. A.3 EXPONENTIAL DELAY DISTRIBUTION.We consider delays drawn from geometric distribution (GD), which is the discrete version of exponential distribution. For each iterate we randomly select a worker to be the straggler with large mean delay (p = 0.1), while all other non-straggler workers have small delays. The non-straggler delay is drawn from GD with p chosen to achieve the same mean delay as in the uniform case (after factoring in straggler) in the main text. The delay is drawn per worker for each iteration, and thus a straggler's outgoing updates to all workers suffer the same delay. FIG3 (c) in the main text shows the convergence speed under the corresponding staleness s with the same mean delay (though s is not a parameter in GD). It exhibits trends analogous to FIG1 in the main text: staleness slows convergence substantially and overall impacts deeper networks more. We present additional for DNNs. Fig. 6 shows the number of batches, normalized by s = 0, to reach convergence using 1 hidden layer and 1 worker under varying staleness levels and batch sizes. Overall, the effect of batch size is relatively small except in high staleness regime (s = 32). Fig. 7 shows the number of batches to reach convergence, normalized by s = 0 case, for 5 variants of SGD using 1 worker. The are in line with the analyses in the main text: staleness generally leads to larger slow down for deeper networks than shallower ones. SGD and Adagrad are more robust to staleness than Adam, RMSProp, and SGD with momentum. In particular, RMSProp exhibit high variance in batches to convergence (not shown in the normalized plot) and thus does not exhibit consistent trend. The are consistent with the observations and analyses in the main text, namely, that having more workers amplifies the effect of staleness. We can also observe that SGDS is more robust to staleness than Adam, and shallower networks are less impacted by staleness. In particular, note that staleness sometimes accelerates convergence, such as in FIG6. This is due to the implicit momentum created by staleness.A.5 LDA AND ADDITIONAL FOR LDAIn LDA each token w ij (j-th token in the i-th document) is assigned with a latent topic z ij from totally K topics. We use Gibbs sampling to infer the topic assignments z ij. The Gibbs sampling step involves three sets of parameters, known as sufficient statistics: FORMULA7 Figure 6: The number of batches to reach 95% test accuracy using 1 hidden layer and 1 worker, respectively normalized by s = 0.vector φ w ∈ R K where φ wk is the number of topic assignments to topic k = 1,..., K for word (vocabulary) w across all documents;φ ∈ R K whereφ k = W w=1 φ wk is the number of tokens in the corpus assigned to topic k. The corpus (w ij, z ij) is partitioned to workers, while φ w andφ are shared model parameters. We measure the model quality using log likelihood. We present additional of LDA under different numbers of workers and topics in FIG12 and FIG1. These panels extends Fig. 3(c)(d) in the main text. See the main text for experimental setup and analyses and experimental setup. We show the convergence curves for MF under different numbers of workers and staleness levels in FIG1. It is evident that higher staleness leads to a higher variance in convergence. Furthermore, the number of workers also affects variance, given the same staleness level. For example, MF with 4 workers incurs very low standard deviation up to staleness 20. In contrast, MF with 8 workers already exhibits a large variance at staleness 15. The amplification of staleness from increasing number of Figure 7: The number of batches to reach 92% test accuracy using DNNs with varying numbers of hidden layers under 1 worker. We consider several variants of SGD algorithms (a)-(e). Note that with depth 0 the model reduces to MLR, which is convex. The numbers are averaged over 5 randomized runs. We omit the whenever convergence is not achieved within the experiment horizon (77824 batches), such as SGD with momentum at depth 6 and s = 32. Recurrent Neural Networks (RNNs) are widely used in recent natural language processing tasks. We consider long short-term memory (LSTM) BID18 applied to the language modeling task, using a subset of Penn Treebank dataset (PTB) BID34 containing 5855 words. The dataset is pre-processed by standard de-capitalization and tokenization. We evaluate the impact of staleness for LSTM with 1 to 4 layers, with 256 neurons in each layer. The maximum length for each sentence is 25. Note that 4 layer LSTM is about 4x more model parameters than the 1 layer LSTM, which is the same ratio between ResNet32 and Resnet 8. We use batch size 32 similar to other experiments. We consider staleness s = 0, 4, 8, 16 on 8 workers. The model quality is measured in perplexity. FIG1 shows the number of batches needed to reach the desired model quality for RNNs with varying network depths. We again observe that staleness impacts deeper network variants more than shallower counterparts, which is consistent with our observation in CNNs and DNNs.

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.