Question: represent the text to find a very short summary (1-2 sentences max)

Question: Existing works in deep Multi-Agent Reinforcement Learning (MARL) mainly focus on coordinating cooperative agents to complete certain tasks jointly. However, in many cases of the real world, agents are self-interested such as employees in a company and clubs in a league. Therefore, the leader, i.e., the manager of the company or the league, needs to provide bonuses to followers for efficient coordination, which we call expensive coordination. The main difficulties of expensive coordination are that i) the leader has to consider the long-term effect and predict the followers' behaviors when assigning bonuses and ii) the complex interactions between followers make the training process hard to converge, especially when the leader's policy changes with time. In this work, we address this problem through an event-based deep RL approach. Our main contributions are threefold. We model the leader's decision-making process as a semi-Markov Decision Process and propose a novel multi-agent event-based policy gradient to learn the leader's long-term policy. We exploit the leader-follower consistency scheme to design a follower-aware module and a follower-specific attention module to predict the followers' behaviors and make accurate response to their behaviors. We propose an action abstraction-based policy gradient algorithm to reduce the followers' decision space and thus accelerate the training process of followers. Experiments in resource collections, navigation, and the predator-prey game reveal that our approach outperforms the state-of-the-art methods dramatically. Deep Multi-Agent Reinforcement Learning (MARL) has been widely used in coordinating cooperative agents to jointly complete certain tasks where the agent is assumed to be selfless (fully cooperative), i.e., the agent is willing to sacrifice itself to maximize the team reward. However, in many cases of the real world, the agents are self-interested, such as taxi drivers in a taxi company (fleets) and clubs in a league. For instance, in the example of taxi fleets , drivers may prefer to stay in the area with high customer demand to gain more reward. It is unfair and not efficient to compel the taxi driver to selflessly contribute to the company, e.g., to stay in the low customer demand area. Forcing the drivers to selflessly contribute may increase the income for the company in a short-term but it will finally causes the low efficient and unsustainable of that company in the long run because the unsatisfied drivers may be demotivated and even leave the company. Another important example is that the government wants some companies to invest on the poverty area to achieve the fairness of the society, which may inevitably reduce the profits of companies. Similar to previous example, the companies may leave when the government forces them to invest. A better way to achieve coordination among followers and achieve the leader's goals is that the manager of the company or the government needs to provide bonuses to followers, like the taxi company pays extra bonuses for serving the customers in rural areas and the government provides subsidies for investing in the poverty areas, which we term as expensive coordination. In this paper, we solve the large-scale sequential expensive coordination problem with a novel RL training scheme. There are several lines of works related to the expensive coordination problem, including mechanism design and the principal-agent model . However, these works focus more on static decisions (each agent only makes a single decision). To consider sequential decisions, the leader-follower MDP game (; 2016) and the RL-based mechanism design are introduced but most of their works only focus on matrix games or small-scale Markov games, which cannot be applied to the case with the large-scale action or state space. The most related work is M 3 RL where the leader assigns goals and bonuses by using a simple attention mechanism (summing/averaging the features together) and mind (behaviors) tracking to predict the followers' behaviors and makes response to the followers' behaviors. But they only consider the rule-based followers, i.e., followers with fixed preference, and ignore the followers' behaviors responding to the leader's policy, which significantly simplifies the problem and leads the unreasonability of the model. In the expensive coordination problem, there are two critical issues which should be considered: 1) the leader's long-term decision process where the leader has to consider both the long-term effect of itself and long-term behaviors of the followers when determining his action to incentivise the coordination among followers, which is not considered in ; and 2) the complex interactions between the leader and followers where the followers will adapt their policies to maximize their own utility given the leader's policy, which makes the training process unstable and hard, if not unable, to converge in large-scale environment, especially when the leader changes his actions frequently, which is ignored by . In this work, we address these two issues in the expensive coordination problem through an abstraction-based deep RL approach. Our main contributions are threefold. We model the leader's decision-making process as a semiMarkov Decision Process (semi-MDP) and propose a novel event-based policy gradient to learn the leader's policy considering the long-term effect (leader takes actions at important points rather than at each step to avoid myopic decisions.) (Section 4.1). A well-performing leader's policy is also highly dependent on how well the leader knows the followers. To predict the followers' behaviors precisely, we show the leader-follower consistency scheme. Based on the scheme, the follower-aware module, the follower-specific attention module, and the sequential decision module are proposed to capture these followers' behaviors and make accurate response to their behaviors (Section 4.2). To accelerate the training process, we propose an action abstraction-based policy gradient algorithm for the followers. This approach is able to reduce followers' decision space and thus simplifies the interaction between the leader and followers as well as accelerates the training process of followers (Section 4.3). Experiments in resource collections, navigation and predatorprey show that our method outperforms the state-of-the-art methods dramatically. Our works are closely related to leader-follower RL, temporal abstraction RL, and event-based RL. Leader-follower RL. The leader-follower RL targets at addressing the issue of expensive coordination where the leader wants to maximize the social benefit (or the leader's self-benefit) by coordinating non-cooperative followers through providing them bonuses. Previous works have investigated different approaches to solve the expensive coordination, including the vanilla leader-follower MARL (; Laumônier &), leader semi-MDP , multiple followers and sub-followers MARL , followers abstraction , and Bayesian optimization . But most of them focus on simple tabular games or small-scale Markov games. The most related work leverages the deep RL approach to compute the leader's policy of assigning goals Figure 1: Overview of our framework. The details of the leader's module and the follower's module can be found in Section 4.2 and Section 4.3, respectively. The implement details of each module can be found in Appendix D.2.1. and bonuses to rule-based followers. But their method performs poorly when the followers are RLbased. In this work, we aim to compute the leader's policy against the RL-based followers in the complex and sequential scenarios. Temporal abstraction RL. Our methods are also related to temporal abstraction method (; ; ; ; ;). The basic idea of temporal abstraction is to divide the original one-level decision process into a two-level decision process where the high-level part is to decide the meta goal while the low-level policy is to select the primitive actions. Our leader's decision process is different from those methods mentioned above because the leader's policy can naturally form as an intermittent (temporal abstraction) decision process (semi-MDP) and it is unnecessary to design the two-level decision process for the leader (since the low-level decision process is the follower). Based on the nature of the leader, a novel training method is introduced. Event-based RL & Planning. Previous studies also focus on using events to capture important elements (e.g., whether agent reaches a goal) during the whole episode. regard the leader's action and the environment feedback as events in the continuous time environment.; leverage events to capture the fact that an agent has accomplished some goals. We adopt this idea by depicting the event as the actions taken by the leader at some time steps and design a novel event-based policy gradient to learn the long-term leader's policy. Our research focuses on single-leader multi-follower Stackelberg Markov Games (SMG) , which can be formulated as a tuple G = N, S, A, Ω, P, R, γ. N is the set of N followers, i.e., |N | = N. S is the set of states. s 0 ∈ S 0 ⊂ S is an initial state and S 0 is the set of initial states. A = × k∈N A k is the set of joint actions for followers where a k ∈ A k is an action for the k-th follower. ω ∈ Ω = × k∈N Ω k is an action for the leader and k is a goal and a bonus that the leader assigns to the k-th follower. P: S × A → ∆(S) is the transition function 1 and R = × k∈N r k × r l is the reward function set where r k: S × A × Ω → R is the reward function for the k-th follower and r l: S × A × Ω → R is the reward function for the leader. γ is the discount factor and a is a joint action of followers. The leader's policy is defined as µ = µ k k∈N where µ k: Ω × S → ∆(Ω k) is the leader's action to the k-th follower given the leader's action in the previous timestep ω t−1 and the current state s t. ∆(·) is a probability distribution. The followers' joint policy is defined as π = π k where is the k-th follower policy given the leader's action ω k t and the current state s t. Given the policy profile of the leader and followers µ, π, the follower's utility is defined as at, ωt) and the leader's utility is J(µ, π) = E T t=0 γ t r l t (st, at, ωt). We assume that the leader and followers aim to maximize their own utilities. We define the trajectory τ as a sequence of state, leader's action, and followers' actions ω −1, (s t, a t, ω t) T t=0 where ω −1 is the first step leader's action and is set to zero. (a) A simple example for the illustration of AT. Suppose that the whole step is 4, the AT = {e (b) The probabilistic graphical model of the proposed framework. Dotted line means that β affects the final of ω indirectly. ω−1 is set to be zero. In this section, we propose a novel training scheme to train a well-performing leader policy against both rule-based and RL-based followers in the expensive coordination problem. We address the two issues, the leader's long-term decision process and the complex interactions between the leader and followers, with three key steps: (a) we model the leader's decision-making process as a semiMarkov Decision Process (semi-MDP) and propose a novel event-based policy gradient to take actions only at important time steps to avoid myopic policy; (b) to accurately predict followers' behaviors, we construct a follower-aware module based on the leader-follower consistency, including a novel follower-specific attention mechanism, and a sequential decision module to predict followers' behaviors precisely and make accurate response to these behaviors; and (c) an action abstractionbased policy gradient method for followers is proposed to simplify the decision process for the followers and thus simplify the interaction between leader and followers, and accelerate the convergence of the training process. We first describe the event-based trajectory optimization for the leader. As we mentioned above, the leader's decision process can be naturally formulated as a semi-MDP . Therefore, we firstly describe the basic ideas of semi-MDP using the modified option structure. We define the modified option as a tuple: where µ is the leader's policy as we defined above and β k (s t, ω t−1): S × Ω → is the termination function for the k-th follower, to indicate the probability whether the leader's action to the k-th follower changes (ω k t−1 = ω k t). Based on these definitions, we formulate the one-step option-state transition function with decay as: . Differently, we do not have the low-level policy here (the low-level policy is the follower) and since we only focus on the finite time horizon, γ is set to be 1. Our modified option is used to depict the long-term decision process for the leader as shown in Fig. 2. Now we start to discuss our leader's policy gradient. In fact, it is not easy to directly optimize the leader's utility based on this multi-agent option-state transition function since this form includes leader's different action stages to different followers. Notice that for a sampled trajectory, the occurrence of the leader actions is deterministic. Therefore, we can regard the time step and the action the leader takes at that step as an event and define the (universal) event set We use the notation e k i = t i, ω k ti to represent the leader's action to the k-th follower at step t i, i is the index of the event. Since we focus on the change of the actions from the leader, we further define a set that represents a collection of new actions (ω k t = ω k t−1) taken by the leader within that trajectory: where t i − 1 is the previous time step. A T represents when and how the leader commits to a new action (an example can be found in Fig. 2a). For brevity, e k j ∈ A T means e k j ∈ U T \A T. The probability of A T can be represented as: where t j − 1 is the previous time step for t j. This equation illustrates that the probability of the occurrence of a certain leader's event set within a trajectory. Concretely, the leader changes action to the k-th follower at t i ∈ e k i while maintaining the same action within the interval from t i − 1 ∈ e k i−1. Similarly, we can further define the probability of the whole trajectory τ as: Comparing with P (A T), P (τ) includes the probability of the followers as well as the state transition. Do note that our goal is to maximize max, indicating that the leader is required to select an action that can maximize the accumulated reward, where R τ (T) = T t=0 γ t r l t is the accumulated reward and τ is to stress that its accumulated reward is from the trajectory τ. Following the REINFORCE trick , the policy gradient for the termination function and the leader's policy function can be formulated under the following proposition: Proposition 1. The policy gradients for the termination function β k (s ti, ω ti) and leader's policy function µ k ω k ti |s ti, ω ti−1 can be written as: where θ and ϑ are the parameters for the termination function β k θ and leader's policy µ k ϑ. I(·) and I (·) are the piece-wise functions: All the proofs can be found in Appendix A. Proposition 1 implies that under the event-based method, whether the leader's commitment to a new action will induce different policy gradients for both termination function and the policy function. However, from the empirical , we find that the leader's policy function updates rarely during the whole episode because the policy only updates when the leader commits to a new action, which causes the sample inefficiency. Notice that in fact the leader commits to the same action when e k i / ∈ A T. Therefore, the policy indication function I (·) can be formulated in an alternative way: This form considers both committing to a new action and maintaining the same actions (Details can be found in Remark 2), which we call the Event-Based Policy Gradient (EBPG) and the previous one as the sparse EBPG respectively. Intuitively, the dense EBPG is better than the sparse EBPG because it updates the leader's policy function more frequently than the sparse one. For example, in time step t, supposing that the leader chooses a wrong action for follower k and receives a negative reward. Then, the leader should learn to diminish the action chosen that state by EBPG. The sparse EBPG only do one PG during before terminating the action (at the committing action step) while the dense one does PG in each step before terminating the action. The latter can provide more signal to correct the wrong action. Experiments also reveal that the dense one is better (Sec. D.3.3). The EBPG approach is able to improve leader's performance. However, it is still very hard for the leader to choose actions considering long-term effect only based on the current state information. This is because the followers change their behaviors over time according to the leader's policy. Therefore, we introduce new modules and training schemes so as to capture the change of the followers' behaviors as well as the global state. To abstract the complicated state information, we use neural networks to learn the state representation. To capture the followers' behaviors and make accurate response to their behaviors, we design three modules: we exploit the leader-follower consistency under game regularization and policy bound conditions, based on the consistency, a follower-aware module is introduced and based on the follower-aware module, a novel attention mechanism, and sequential decision making module is designed to make accurate response to these followers' behaviors as shown in Fig. 1. Leader-Follower Consistency. In previous works, a surge of researches focus on predicting other agents' behaviors through historical information, where the other agents are assumed to be opponents of that agent, which is only suitable for zero-sum games (; ;). However, these methods cannot be directly applied to our case because SMG is not zero-sum. We note that This assumption is inspired by . We only extend it into the multi-agent forms. This assumption indicates that the action and states space should be limited and the reward function for the leader action should be smooth. Assumption 2. (Policy Bound) For any agent k, reward function r k and policy is consistency, i.e., This assumption is inspired by . π −k indicates the joint policy without the k-th agent's. This assumption indicates that the change of the leader causes only slightly changes on each followers policy. Based on these two assumptions, we propose a proposition here: Proposition 2. (Leader-Follower Consistency.) If both the assumptions of game regularization and policy bound are satisfied, for ∀ > 0, k ∈ N, there exists δ > 0, such that |µ − µ | ≤ implies π k − π k ≤ δ, where µ and π k are the new policies for the leader and the k-th follower respectively. These methods mentioned above are fully implemented can enhance the performance dramatically. But when facing the RL-based followers, the SMG is still hard to converge. This is because in SMG, the policies of the leader and followers are always changing depending on other agents' performance. To guarantee convergence, the leader can only update its policy when the followers reach (or are near to) the best response policy . However, when the followers are RL-based agents, there is no way to ensure the followers' policies are (near) the best response policies in large-scale SMG and the commonly-seen idea is to provide enough training time but it is unbearable in practice due to the limitation of computing power . To accelerate the training process, inspired by the action abstraction approach which is commonlyseen in Poker and action abstraction RL , we collect the followers' primitive actions sharing the same properties together as a meta policy. Then, the followers only need to select the meta action to make a decision. Therefore, the original game is converted into a meta game, which is easy to solve. Specifically, we define the policy for the k-th follower as: k is the augmented state for the follower (the combination of current state and the leader's action). π k meta (z|ŝ) is the meta policy for the k-th follower and z is the high-level (meta) action. We hypothesize that the lower-level policy (the policy to choose the primitive actions) is already known (rule-based) and deterministic, i.e., π k lower (a k |ŝ, z) = 1. For instance, given the example of the navigation task, the π k meta can be the selection to which landmark to explore while π k lower is a specific route planning algorithm (such as Dijkstra Algorithm). Based on this assumption, we can design a novel policy gradient to train the meta policy: where λ k is the parameter for meta-policy π k meta (Details can be found in Lemma 3). In this section, we discuss how to design the leader's and followers' loss functions. Loss Functions for the Leaders. The basic structure for the leader is the actor-critic structure . We find that adding regularizers can enhance the leader's performance and we implement the maximum entropy for the leader's policy function as well as the L2 regularization for the termination function, i.e., ) and L reg = β 2. We also use imitation learning to learn the predicted action function p k. Following the same logic of , two baseline functions φ g (c t) and φ b (c t) are also introduced to further reduce the variance. Details can be found in Appendix B. Loss Functions for the RL-Based Followers. The basic structure for each follower is also based on the actor-critic structure. We leverage the action abstraction policy gradient as we mentioned above. The learning rate between the leader and follower should satisfy the two time-scale principle (Roughly speaking, the leader learns slower than the follower(s)), similar to . Details can be found in Appendix B and the pseudo-code can be found in Appendix C. Resource Collection Multi-bonus Resource Collection Navigation Predator-prey We evaluate the following tasks to testify the performance of our proposed method. All of these tasks are based on SMG mentioned above. resource collections: each follower collects three types of resources including its preferred one and the leader can choose two bonuses levels ; multi-bonus resources collections: based on, the leader can choose four bonuses levels; modified navigation: followers are required to navigate some landmarks and after one of the landmarks is reached, the reached landmark disappears and new landmark will appear randomly. modified predator-prey: followers are required to capture some randomly moving preys, prizes will be given after touching them. Both and are based on and we modify them into our SMG setting. Moreover, to increase the difficulty, in each episode, the combinations of the followers will change, i.e., in each task, there are 40 different followers and at each episode, we randomly choose some followers to play the game. More details can be found in Appendix D. Baselines & Ablations. To evaluate our method, we compare a recently proposed method as our baseline: M 3 RL . We do not include other baselines because other methods cannot be used in our problems, as justified in . For the ablations of the leader part, we choose: ours: the full implementation of our method. ours w/o EBPG: removing the event-based policy gradient part; ours w/o Attention: replacing follower-specified attention model by the original attention model mentioned in . For the follower part, we choose (a) with rule-based follower (b) with vanilla RL-based follower, and (c) with action abstraction RL-based follower to testify the ability of our methods when facing different followers. Hyper-Parameters. Our code is implemented in Pytorch . If no special mention, the batch size is 1 (online learning). Similar to , we set the learning rate as 0.001 for the leader's critic and followers while 0.0003 for the leader's policy. The optimization algorithm is Adam . Our method takes less than two days to train on a NVIDIA Geforce GTX 1080Ti GPU in each experiment. For the loss function, we set the λ 1 = 0.01 and λ 2 = 0.001. The total training episode is 250, 000 for all the tasks (including both the rule-based followers and the RL-based followers). To encourage exploration, we use the ι-greedy 2. For the leader, the exploration rate is set to 0.1 and slightly decreases to zero (5000 episode). For the followers, the exploration rate for each agent is always 0.3 (except for the noise experiments). The quantitative with different tasks are shown in Figs. 3 & 4. For the rule-based followers, from Fig. 3, we find that our method outperforms the state-of-the-art method in all the tasks, showing that our method is sample efficient and fast to coverage. There is an interesting phenomenon that in the task of multi-bonus resource collections and navigation, only our method obtains a positive reward, indicating that our method can work well in complicated environments. For ablations, we can see that ours w/o attention and ours w/o EBPG are worse than ours, representing these components do enhance the performance. For the RL-based followers, from Fig. 4, we observe that when facing the RL-based method with action abstraction, our approach outperforms the baseline method in all the tasks (in predator-prey game, the reward for ours is twice as that of the state-of-the-art). We also find that without action abstraction, the reward is less than zero, revealing that the abstraction does play a crucial role in stabilizing training. This experiment is to evaluate whether our method is robust to the noise, i.e., the follower randomly takes actions. We make this experiment by introducing noise into the follower decision. From Table 1, we can find that our method reaches a higher total reward (more than 5) among all the environment with noise than the state-of-the-art, indicating that our method is robust to the noise. We also observe that the total reward for the baseline method becomes lower with the increase of the noise while our method is more robust to the change. Moreover, for the incentive (the total gain), we find that our method gains much more incentive than the state-of-the-art method, showing that our method coordinates have a better coordination the followers than the state-of-the-art method. We also do a substantial number of experiments. However, due to the space limitation, we can only provide some here: The total incentives: incentive can reveal the performance of successful rate interacting with the followers. Our method outperforms the state-of-the-art method, indicating that our method has a better ability to interact with the followers. Sparse EBPG: we compare the performance gap between sparse EBPG and (dense) EBPG. This show that the sparse one is worse than the dense one, supporting the assumption that the dense signal can improve the sample efficiency. Visualizing attention: We visualize the attention module to find what it actually learns and the indicates that our attention mechanism does capture the followers whom the leader needs to assign bonuses to. Two time-scale training: We testify whether our two timescale training scheme works and the ablation shows that this scheme does play an important role in improving the performance of both the leader and the followers. The committing interval: We observe that the dynamic committing interval (our method) performs better than the one with fixed committing intervals. Reward for RL-based followers: we show the reward for the followers, which can provide the situation of the followers. The represents that our method aids the followers to gain more than the state-of-the-art method. Number of RL-based followers: finally, we testify our method in cases with different number of RL-based followers. The shows that our method always performs well. The full can be found in Appendix D. This paper proposes a novel RL training scheme for Stackelberg Markov Games with single leader and multiple self-interested followers, which considers the leader's long-term decision process and complicated interaction between followers with three contributions. 1) To consider the long-term effect of the leader's behavior, we develop an event-based policy gradient for the leader's policy. 2) To predict the followers' behaviors and make accurate response to their behaviors, we exploit the leader-follower consistency to design a novel follower-aware module and follower-specific attention mechanism. 3) We propose an action abstraction-based policy gradient algorithm to accelerate the training process of followers. Experiments in resource collections, navigation, and predator-prey game reveal that our method outperforms the state-of-the-art methods dramatically. We are willing to highlight that SMGs contribute to the RL (especially MARL) community with three key aspects: 1). As we mentioned in the Introduction, most of the existing MARL methods assume that all the agents are willing to sacrifice themselves to maximize the total rewards, which is not true in many real-world non-cooperative scenarios. On the contrary, our proposed method realistically assumes that agents are self-interested. Thus, SMGs provide a new scheme focusing more on the self-interested agents. We think this aspect is the most significant contribution to the RL community. 2). The SMGs can be regarded as the multi-agent system with different roles (the leader and the followers) and our method provides a solution to that problem. 3). Our methods also contribute to the hierarchical RL, i.e., it provides a non-cooperative training scheme between the high-level policy (the leaders) and the low-level policy (the followers), which plays an important role when the followers are self-interested. Moreover, our EBPG also propose an novel policy gradient method for the temporal abstraction structure. There are several directions we would like to investigate to further extend our SMG model: i) we will consider multiple cooperative/competitive leaders and multiple self-interested followers, which is the case in the labor market, ii) we will consider multi-level leaders, which is the case in the hierarchical organizations and companies and iii) we will consider the adversarial attacks to our SMG model, which may induce extra cost to the leader for efficient coordination. We believe that our work is a preliminary step towards a deeper understanding of the leader-follower scheme in both research and the application to society. We would like to thank Tianming Shu, Suming Yu, Enrique Munoz de Cote, and Xu He for their kind suggestions and helps. We also appreciate the anonymous reviewers for their hard work. where θ and ϑ are the parameters for the termination function β k θ and the leader's policy µ k ϑ. I(·) and I (·) are the piece-wise functions: Proof. First recall the utility for the leader: Where m is the number of the times taking new action, m ≤ T. If m = T, implying that the leader has taken new action at each time. P(|A T | = m) means the probability of times taking new action within an episode. Take derivatives on both LHS and RHS, we get: For brevity, we use P (τ) to represent P (s 0) k∈N P (s t+1 |s t, a t), the trajectory probability. And we use i ∈ e k i and j ∈ e k j to represent i ∈ e k i ∈ A T to j ∈ e k j / ∈ A T with a slight abuse of notation. Thus the equation mentioned above can be further simplified as: The equation above is exactly the REINFORCE trick and the rule of derivations. The approximation indicates that one trajectory only has one A T 3. Also based on the definition of e k i and e k j, the equation can be rewritten in a more compact form: Where I(·) is the piece-wise function: This is the first part of the proof (the policy gradient for the termination function). Here, we start proving the second part (the policy gradient for the leader's action). The proof of the second part is similar to the first part. We rewrite it to a more compact form: Remark 1. Some researches also focus on event-based RL but either on single-agent continuous time or reward representation . We are the first to develop and implement the event-based policy gradient into the multi-agent system. Remark 2. In fact, the policy gradient for the leader actions might be somewhat sparse, i.e., we only update the policy when the leader changes its actions. Notice that the leader commits to the same action when e k i / ∈ A T. Therefore, the probability of leader's action P (A T) can also represented as: Then the policy gradient for leader's policy, where For any agent k, the corresponding reward function r k w.r.t ω is CLipschitz continuous. Where C = (1 − γ) −1 C.The last equation is drawn form the Assumption 1 and the inequality of a geometric series: Proof. By combining Lemma 2 and Assumption 2, we can draw that: If there exist |ω − ω | < and we have: And we set δ ≥ (1 − γ)C, the consistency is established. Lemma 3. (Action Abstraction Policy Gradient.) Under the assumption that the low-level follower policy π When π k lower (a|ŝ, z) is fixed and deterministic, the equation can be: lower (a t |ŝ t, z t) = 1 is the partition function. For brevity, with a slight abuse of notation, we omit the superscript for variables a and z which represents the index of an agent. We add a baseline function to reduce the variance of the event-based policy gradient for the leader. We adopt the idea of successor representation as two expected baseline functions:φ g (c t) andφ b (c t). For the gain baseline function: For the bonus-based baseline function: Two baseline neural network functions with parametersθ g andθ b are trained through minimizing the mean square error: Where c t is the attention-based latent variable. To this end, the gradient for the leader can be formulated as: Where are the baseline policy gradients. We also leverage the imitation learning to learn the action probability function p k a The illustration of experimental scenarios can be found in Figure 5. Here we give some details about these environments: Resource Collections. This task is similar to , which is based on the scene that the leader and the followers collect some resources. Each follower has its own preference which might be the same (or against) to the leader's preference. In order to make the followers obey the leader's instruction, the leader should pay the followers bonuses. There are total 4 types of resources and for different resources each agent has different preferences. The leader owns two type of bonus (1 or 2) and 4 types of goals (each resource is a goal). The number of leader is 1 while the number of followers is 4. Multi-Bonus Resource Collections. This task is similar to Resource Collections. Except that the leader can take 4 level bonuses (a bonus from 1 to 4) while each agent owns one skill. The number of leader is 1 while the number of followers is 4. Modified Navigation. This task is original from . We make some modifications here to make it suitable our SMG: the leader and the followers are going to navigate some landmarks. Each follower has its own preference which might be the same (or against) to the leader's preference. When a landmark has been navigated, it disappears immediately and a new landmark will appear. There are total 6 types of landmarks and for different landmarks, each agent has different preferences. The leader owns two type of bonuses and 6 types of goals (each landmark is a goal). The number of leader is 1 while the number of followers is 8 and the number of the landmarks is 6. Modified Predator-Prey. This task is also original from . We make some modification here to make it suitable our SMG: the leader and the followers are going to catch some preys. Each follower has its own preference which might be the same (or against) to the leader's preference. In each step, whether a prey has been caught, it randomly chooses a direction to go. Catching a prey will not make it disappear, which means that the preys can exist until the game ends. There are total 8 types of preys and for different preys, each agent has different preferences. The leader owns two type of bonuses and 8 types of goals (each prey is a goal). The followers are faster than the preys. The number of leader is 1 while the number of followers is 10 and the number of the landmarks is 8. Reward Design. The rewards mentioned in Section 3 are the general forms. Here, we define two specified forms of the leader and followers reward function in our experiments: Leader Reward. We define v g as the prize (utility) for finishing task g. We set the reward function for the leader at step t as: r emphasize that our leader reward is total different from the : in their approaches, the leader changes its mind after signing a contract will not be punished. To make it suitable to the real world, we modify the reward as the leader should pay the followers bonuses immediately after signing the contract and cannot get back if it gives up the contract. Follower Reward. For the followers, we set the reward for the k-th follower as: where u k,g reveals the payoff of the k-th follower when finishing task g (the preference). Specifically, r k t indicates that the follower can either follow the leader's instruction or just do what it prefers to. The followers will receive reward immediately after signing the contract (the leader and the followers achieve an agreement). I s k t = s g means that the follower finishes the task g at step t. A penalty is added to the followers if the followers betray the leader (the followers and the leader sign the contract but the followers do not agree to work). Our network is based on . Some do not suit our method. We do some modification here: We change the vanilla attention mechanism (sum/average all the history and action of each follower together) to a follower-specified one: each follower has a weight which indicates how important the follower is at the current step. The output for g and b are changed into the sequential form, i.e., we first calculate p(g k t can reveal how well the leader interacts with the followers; the higher the R in is, the more successful the coordination between the leader and the followers. From Figure 6 and 7, comparing with the state-of-the-art method, we can see that our method far outperforms the state-of-the-art method, which reveals that our method does have a better ability to coordinate with the followers. In all of the scenarios, without the EBPG, the performance of our method is worse than ours with EBPG. Specifically, in some scenarios (e.g., multi-bonus resource collections, navigation), without the EBPG, the performance of our method is (or nearly) similar to the performance of M 3 RL, showing the effectiveness of our novel policy gradient. Moreover, we can notice that in navigation environment, without follower-specified attention, the performance of our method diminishes rapidly, which implies that in some scenarios, attention does play an important role. Figure 9: The ablation study of sparse EBPG in the predator-prey task. In this section, we are going to testify the robust of our method. Specifically, we evaluate whether our method is robust to the noise. We make this experiment by introducing noise into the follower decision. For example, if we set the noise function as 30%, indicating that there is 30% probability that the followers will choose action randomly. This experiment testifies whether the dense event-based policy gradient increases the leader's performance comparing with the sparse event-based policy gradient. We make ablations here: Ours: the full structure of our method; Sparse Event-Based Policy Gradient (sparse EBPG): the fully structure of ours except that the EBPG is replaced by sparse event-based policy gradient; sparse EBPG w/o attention: replacing the follower-specified attention mechanism by averaging the input features. From Figure 8 and 9 we can find that if the policy gradient is sparse, its performance is worse than the dense one, implying that the dense method does improve the leader's performance. There is also an interesting phenomenon that sparse EBPG with follower-specified attention mechanism performs better than that without, revealing that the attention can stabilize training when the training signal is sparse. Figure 11: The ablation study of reward curves for two time-scale update method in resource collections task. Following the same logic of , we visualize the weight of the attention when the leader takes actions. From Figure 10, we find that the attention mechanism does learn to strongly attend to the followers that the leader needs to take actions. The followers with leader's commitment obtain much higher attention weight than others, showing that the attention module actually learn to identify the important followers while leader committing new action. Thus, the attention mechanism does play an important role in improving the performance. In order to evaluate the performance of our two time-scale update scheme (TTSU), we do an ablation study as shown in Fig 11. We can find that the performance where the followers' learning rate α (1 × 10 −3) is much larger than the leader's β (3 × 10 −4) is better than the performance where the leader's learning rate is similar to the followers (1 × 10 −3). Moreover, without TTSU, the reward curves of training methods become unstable, revealing that TTSU can stabilize the training process. In fact, TTSU improves the rate of convergence and play an important role in improving performance. We evaluate the leader's performance between static committing interval and our dynamic committing interval. As shown in Figure 12, we observe that all the different fixed committing intervals only change the rate of convergence and do not enhance the leader's performance. All the fixed committing intervals are much worse than our dynamic committing approach, revealing the fact that our dynamic committing approach aids a lot in improving the leader's performance. We are interesting in the reward for the RL-based follower(s). Intuitively, a well-performing leader can make the follower gain more. As shown in Figure 13, the reward for RL follower is higher than M 3 RL follower in all the tasks. This represents the leader can coordinate the followers better and make them gain more reward than other methods, which forms a win-win strategy. Finally, we evaluate the leader's performance with different number of RL-based followers. As shown in Figure 15, we find that our method outperforms the state-of-the-art method when facing different number of RL-based workers. To further illustrate the performance of different combinations of our methods, we make an extra ablation here. We choose the resource collections as the environments. Our analysis is as follows: For the RL based followers scenario, As shown in Table 2, we find that the action abstraction policy gradient is very important to converge. Additionally, adding different modules can improve the performance and the method with all the modules reach the highest reward than other combinations. The contribution ranking for each module is: Action abstraction policy gradient > EBPG > Leaderfollower consistency.

represent the tldr for finding the corresponding scientific article:

We propose an event-based policy gradient  to train the leader and an action abstraction policy gradient to train the followers in leader-follower Markov game.

represent this text to find a much much longer article about it with say hundreds-thousands of words
------
Fewshot example: "To our knowledge, this is the first study to show how neural representations of space, including grid-like cells and border cells as observed in the brain, could emerge from training a recurrent neural network to perform navigation tasks." == "Decades of research on the neural code underlying spatial navigation have revealed a diverse set of neural response properties. The Entorhinal Cortex (EC) of the mammalian brain contains a rich set of spatial correlates, including grid cells which encode space using tessellating patterns. However, the mechanisms and functional significance of these spatial representations remain largely mysterious. As a new way to understand these neural representations, we trained recurrent neural networks (RNNs) to perform navigation tasks in 2D arenas based on velocity inputs. Surprisingly, we find that grid-like spatial response patterns emerge in trained networks, along with units that exhibit other spatial correlates, including border cells and band-like cells. All these different functional types of neurons have been observed experimentally. The order of the emergence of grid-like and border cells is also consistent with observations from developmental studies. Together, our suggest that grid cells, border cells and others as observed in EC may be a natural solution for representing space efficiently given the predominant recurrent connections in the neural circuits. Understanding the neural code in the brain has long been driven by studying feed-forward architectures, starting from Hubel and Wiesel's famous proposal on the origin of orientation selectivity in primary visual cortex BID19. Inspired by the recent development in deep learning BID25 BID30 BID18 BID39, there has been a burst of interest in applying deep feedforward models, in particular convolutional neural networks (CNN) BID29, to study the sensory systems, which hierarchically extract useful features from sensory inputs (see e.g., BID61 ; BID24 ; BID22 ; BID60).For more cognitive tasks, neural systems often need to maintain certain internal representations of relevant variables in the absence of external stimuli-a process that requires more than feature extraction. We will focus on spatial navigation, which typically requires the brain to maintain a representation of self-location and update it according to the animal's movements and landmarks of the environment. Physiological studies done in rodents and other mammals (including humans, non-human primates and bats) have revealed a variety of neural correlates of space in Hippocampus and Entorhinal Cortex (EC), including place cells BID41, grid cells BID10 BID15 BID11 BID62 BID23 BID20, along with border cells BID49, band-like cells BID27 and others (see FIG0). In particular, each grid cell only fires when the animal occupies a distinct set of physical locations, and strikingly these locations lie on a lattice. The study of the neural underpinning of spatial cognition has provided an important window into how high-level cognitive functions are supported in the brain BID0.How might the spatial navigation task be solved using a network of neurons? Recurrent neural networks (RNNs) BID18 BID12 BID43 BID54 BID13 BID53 seem particularly useful for these tasks. Indeed, recurrent-based continuous attractor networks have been one popular type of models proposed for the formation of grid cells BID4 BID5 and place cells BID45. Such models have provided valuable insights into one set of possible mechanisms that could support the formation of the grids. However, these models typically rely on fine-tuned connectivity patterns, in particular the models need a subtle yet systematic asymmetry in the connectivity pattern to move the attractor state according to the animal's own movement. The existence of such a specific 2D connectivity in rodent EC remains unclear. Additionally, previous models have mainly focused on grid cells, while other types of responses that co-exist in the Entorhinal Cortex have been largely ignored. It would be useful to have a unified model that can simultaneously explain different types of neural responses in EC.Motivated by these considerations, here we present an alternative modeling approach for understanding the representation of space in the neural system. Specifically, we trained a RNN to perform some spatial navigation tasks. By leveraging the recent development in RNN training and knowledge of the navigation system in the brain, we show that training a RNN with biologically relevant constraints naturally gives rise to a variety of spatial response profiles as observed in EC, including grid-like responses. To our knowledge, this is the first study to show that grid-like responses could emerge from training a RNN to perform navigation. Our implies that the neural representation in EC may be seen as a natural way for the brain to solve the navigation task efficiently BID55. More generally, it suggests that RNNs can be a powerful tool for understanding the neural mechanisms of certain high-level cognitive functions. recorded when an animal navigates in a square environment, replotted from BID27, with the heat map representing the firing rate of this neuron as a function of the animal's location (red corresponds to high firing rate); a "band-like" cell from BID27; a border cell from BID49; an irregular spatially tuned cell from BID7; a "speed cell" from BID26, which exhibits roughly linear dependence on the rodent's running speed; a "heading direction cell" from BID46, which shows systematic change of firing rate depending on animal's heading direction. b) The network consists of N = 100 recurrently connected units (or neurons) which receive two external inputs, representing the animal's speed and heading direction. The two outputs linearly weight the neurons in the RNN. The goal of training is to make the responses of the two output neurons accurately represent the animal's physical location. c) Typical trajectory after training. As shown, the output of the RNN can accurately, though not perfectly, track the animal's location during navigation. Our network model consists of a set of recurrently connected units (N = 100). The dynamics of each unit in the network u i (t) is governed by the standard continuous-time RNN equation: DISPLAYFORM0 for i = 1,..., N. The activity of each unit, u i (t), is related to the activation of that unit, x i (t), through a nonlinearity which in this study we take to be u i (t) = tanh(x i (t)). Each unit receives input from other units through the recurrent weight matrix W rec and also receives external input, I(t), that enters the network through the weight matrix W in. Each unit has two sources of bias, b i which is learned and ξ i (t) which represents noise intrinsic to the network and is taken to be Gaussian with zero mean and constant variance. The network was simulated using the Euler method for T = 500 timesteps of duration τ /10.To perform a 2D navigation task with the RNN, we linearly combine the firing rates of units in the network to estimate the current location of the animal. The responses of the two linear readout neurons, y 1 (t) and y 2 (t), are given by the following equation: DISPLAYFORM1 The network inputs and outputs were inspired by simple spatial navigation tasks in 2D open environments. The task resembles dead-reckoning (sometimes referred to as path integration), which is ethologically relevant for many animal species BID6 BID38 BID9. To be more specific, the inputs to the network were the animal's speed and direction at each time step. Experimentally, it has been shown that the velocity signals exist in EC BID46 BID26 BID17, and there is also evidence that such signals are necessary for grid formation BID58.Throughout the paper, we adopt the common assumption that the head direction of the animal coincides with the actual moving direction. The outputs were the x-and y-coordinates of the integrated position. The direction of the animal is modeled by modified Brownian motion to increase the probability of straight-runs, in order to be consistent with the typical rodent's behavior in an open environment. The usage of such simple movement statistics has the advantage of having full control of the simulated trajectories. However, for future work it would be very interesting to test the model using different animals' real movement trajectories to see how the might change. Special care is taken when the animal is close to the boundary. The boundary of the environment will affect the statistics of the movement, as the animal cannot cross the boundary. This fact was reflected in the model by re-sampling the angular input variable until the input angle did not lead the animal outside the boundary. In the simulations shown below, the animal always starts from the center of the arena, but we verified that the are insensitive to the starting locations. We optimized the network parameters W rec, W in, b and W out to minimize the squared error in equation between target x-and y-coordinates from a two dimensional navigation task (performed in rectangular, hexagonal, and triangular arenas) and the network outputs generated according to equation. DISPLAYFORM0 Parameters were updated with the Hessian-free algorithm BID35 ) using minibatches of size M = 500 trials. In addition to minimizing the error function in equation we regularized the input and output weights according to equation and the squared firing rates of the units (referred to as metabolic cost) according to equation. In sum, the training aims to minimize a loss function, that consists of the error of the animal, the metabolic cost, and a penalty for large network parameters. DISPLAYFORM1 We find that the are qualitatively insensitive to the initialization schemes used for the recurrent weight matrix W rec. For the presented in this paper, simulations in the hexagonal environment were obtained by initializing the elements of W rec to be zero mean Gaussian random variables with variance 1.5 2 /N, and simulations in the square and triangular environments were initialized with an orthogonal W rec BID48. We initialized the bias b and output weights W out to be zero. The elements of W in were zero mean Gaussian variables with variance 1/N in. We run simulation experiments in arenas with different boundary shapes, including square, triangular and hexagonal. FIG0 shows a typical example of the model performance after training; the network (red trace) accurately tracks the animal's actual path (black). We are mostly interested in what kind of representation the RNN has learned to solve this navigation task, and whether such a representation resembles the response properties of neurons in EC. To test whether the trained RNN developed location-selective representations, we plot individual neurons' mean activity level as a function of the animal's location during spatial exploration. Note that these average response profiles should not be confused with the linear filters typically shown in feedforward networks. Surprisingly, we find neurons in the trained RNN show a range of interesting spatial response profiles. Examination of these response profiles suggests they can be classified into distinct functional types. Importantly, as we will show, these distinct spatial response profiles can be mapped naturally to known physiology in EC. The spatial responses of all units in trained networks are shown in the Appendix. Grid-like responses Most interestingly, we find some of the units in the RNN exhibit clear grid-like responses FIG1 ). These firing patterns typically exhibit multiple firing fields, with each firing field exhibiting roughly circular symmetric or ellipse shape. Furthermore, the firing fields are highly structured, i.e., when combined, are arranged on a regular lattice. Furthermore, the structure of the response lattice depends on the shape of the boundary. In particular, training the network to perform self-localization in a square environment tends to give rectangular grids. In hexagonal and triangular environments, the grids are closer to triangular. Experimentally, it is shown that (medial) EC contains so-called grid cells which exhibit multiple firing fields that lie on a regular grid BID10 BID15. The grid-like firing patterns in our simulation are reminiscent of the grid cells in rodents and other mammals. However, we also notice that the the grid-like model responses typically exhibit few periods, not as many as experimental data (see FIG0 . It is possible that using a larger network might reveal finer grid-patterns in our model. Nonetheless, it is surprising that the gird-like spatial representations can develop in our model, given there is no periodicity in the input. Another potential concern is that, experimentally it is reported that the grids are often on the corners of a triangular lattice BID15 even in square environments (see FIG0, though the grids are somewhat influenced by the shape of the environment. However, the rats in these experiments presumable had spatial experience in other environments with various boundary shapes. Experimentally, it would be interesting to see if grid cells would lie on a square lattice instead if the rats are raised in a single square environment -a situation we are simulating here. Border responses Many neurons in the RNN exhibit selectivity to the boundary FIG1). Typically, they only encode a portion of the boundary, e.g. one piece of wall in a square shaped environment. Such properties are similar to the border cells discovered in rodent EC BID49 BID47 BID31 ). Experimentally, border cells mainly fire along one piece of wall, although some have been observed to fire along multiple borders or along the whole boundary of the environment; interestingly, these multi-border responses were also observed in some RNN models. Currently, it is unclear how the boundary-like response profiles emerge BID49 BID47 BID31. Our model points to the possibility that the border cells may emerge without the presence of tactile cues. Furthermore, it suggests that border cell formation may be related to the movement statistics of the animals, i.e. due to the asymmetry of the movement statistics along the boundary. Band-like responses Interestingly, some neurons in the RNN exhibit band-like responses (FIG1). In most of our simulations, these bands tend to be parallel to one of the boundaries. For some of the units, one of the bands overlaps the boundary, but for others, that is not the case. Experimentally, neurons with periodic-like firing patterns have been recently reported in rodent EC. In one study, it has been reported that a substantial portion of cells in EC exhibit band-like firing characteristics BID27. However, we note that based on the reported data in BID27, the band pattern is not as clear as in our model. Spatially-stable but non-regular responses Besides the units described above, most of the remaining units also exhibit stable spatial responses, but they do not belong to the above categories. These response profiles can exhibit either one large irregular firing field; or multiple circular firing fields, but these firing fields do not show a regular pattern. Experimentally these types of cells have also been observed. In fact, it is recently reported that the non-grid spatial cells constitute a large portion of the neurons in Layer II and III of rodent EC BID7. shown in Figure 3. Interestingly, we observe that the model border cells tend to have almost zero speed-tuning (e.g., see Figure 3g,h).Head direction tuning A substantial portion of the model neurons show direction tuning. There are a diversity of direction tuning profiles, both in terms of the strength of the tuning and their preferred direction. Example tuning curves are shown in Figure 3, and the direction tuning curves of a complete population are shown in the Appendix. Interestingly, in general model neurons which show the strongest head direction tuning do not show a clear spatial firing pattern (see Figure 3a,b,c). This suggests that there are a group of neurons which are mostly responsible for encoding the direction. We also notice that neurons with clear grid-like firing can exhibit a variety of direction tuning strengths, from weak to strong (Figure 3d,e,f). In the Appendix, we quantify the relation between these different tuning properties at the whole population level, which show somewhat complex dependence. Experimentally, the heading direction tuning in EC is well-known, e.g., BID46. Both the grid and non-grid cells in EC exhibit head direction tuning BID46. Furthermore, the linear speed dependence of the model neurons is similar to the properties of speed cells reported recently in EC BID26. Our is also consistent with another recent study reporting that the majority of neurons in EC exhibit some amount of speed tuning BID17.It remains an open question experimentally, at a population level, how different types of tuning characteristics in EC relate to each other. We next investigate how the spatial response profiles evolve as learning/training progresses. We report two main observations. First, neurons that fire selectively along the boundary typically emerge first. Second, the grid-like responses with finer spatial tuning patterns only emerge later in training. For visualization, we perform dimensionality reduction using the t-SNE algorithm BID33. This algorithm embeds 100 model neurons during three phases of training (early, intermediate, and late) into a two-dimensional space according to the similarity of their temporal responses. Here the similarity metric is taken to be firing rate correlation. In this 2D space as shown in FIG3, border cell representations appear early and stably persist through the end of training. Furthermore, early during training all responses are similar to the border related responses. In contrast, grid-like cells typically undergo a substantial change in firing pattern during training before settling into their final grid-like representation FIG3 ).The developmental time line of the grid-like cells and border cells is roughly consistent with developmental studies in rodents. Experimentally, it is known that border cells emerge earlier in development, and they exist at about 2 weeks after the rat is born BID3. The grid cells mature only at about 4 weeks after birth BID28 BID57 BID3. Furthermore, our simulations suggest the reason why border cells emerge earlier in development may be that computationally it is easier to wire-up a network that gives rise to border cell responses.. These cells' response profiles appear early and stably persist through training, illustrated by the short distance they travel in this space. In b), we show that the neurons which eventually become grid cells initially have tuning profiles similar to the border cells but then change their tuning substantially during learning. As a natural consequence, they need to travel a long distance in this space between the early and late phase of the training. Spatial responses are shown for four of these grid-like cells during the late phase of training. We find appropriate regularizations of the RNN to be crucial for the emergence of grid-like representations. We only observed grid-like representations when the network was encouraged to store information while perturbed by noise. This was accomplished by setting the speed input to zero, e.g. zero speed 90% of the time, and adding Gaussian noise to the network (ξ i (t) in equation FORMULA0 ); the precise method for setting the speed input to zero and the value of the noise variance is not crucial for our simulations to develop grid-like representations. The cost function which aims to capture the penalization on the metabolic cost of the neural activity also acts as an important regularization. Our simulations show that the grid-like representation did not emerge without this metabolic cost. In FIG4, we show typical simulation for a square environment, with and without proper metabolic regularization. In the Appendix, we illustrate the effect of regularization further, in particular the role of injecting noise into the RNN units. Our are consistent with the general notion on the importance of incorporating proper constraint for learning useful representations in neural networks BID2. Furthermore, it suggests that, to learn a model with response properties similar to neural systems it may be necessary to incorporate the relevant constraints, e.g., noise and metabolic cost. One natural question is whether the trained RNNs are able to perform localization when the path length exceeds the typical length used during training (500 steps), in particular given that noise in a b Figure 6: Error-correction happens at the boundary and the error is stable over time. At the boundary, the direction is resampled to avoid input velocities that lead to a path extending beyond the boundary of the environment. These changing input statistics at the boundary, termed a boundary interaction, are the only cue the RNN receives about the boundary. We find that the RNN uses the boundary interactions to correct the accumulated error between the true integrated input and its prediction based on the linear readout of equation FORMULA1. Panel a), the mean squared error increases when there are no boundary interactions, but then decreases after a boundary interaction, with more boundary interactions leading to greater error reduction. In the absence of further boundary interaction, the squared error would gradually increase again (blue curve) at roughly a constant rate. b) The network was trained using mini-batches of 500 timesteps but has stable error over a duration at least four orders of magnitude larger. The error of the RNN output (mean and standard deviation shown in black, computed based on 10000 timesteps) is compared to the error that would be achieved by an RNN outputting the best constant values (red).the network would gradually accumulate, leading to a decrease in localization performance. We test this by simulating paths that are several orders of magnitude longer. Somewhat surprisingly, we find the RNNs still perform well (Figure 6b). In fact, the squared error (averaged over every 10000 steps) is stable. The spatial response profiles of individual units also remain stable. This implies that the RNNs have acquired intrinsic error-correction mechanisms during training. As shown earlier, during training some of the RNN units develop boundary-related firing FIG1 ), presumably by exploiting the change of input statistics around the boundary. We hypothesize that boundary interactions may enable error-correction through signals based on these boundary-related activities. Indeed, we find that boundary interactions can dramatically reduce the accumulated error (Figure 6a). Figure 6a shows that, without boundary interactions, on average the squared error grows roughly linearly as expected, however, interactions with the boundaries substantially reduce the error, and more frequent boundary interactions can reduce the error further. Error-correction on grid cells via boundary interactions has been proposed BID16 BID44, however, we emphasize that the model proposed here develops the grid-like responses, boundary responses and the error-correction mechanisms all within the same neural network, thus potentially providing a unifying account of a diverse set of phenomena. In this paper, we trained RNNs to perform path integration (dead-reckoning) in 2D arenas. We found that after training RNNs with appropriate regularization, the model neurons exhibit a variety of spatial and velocity tuning profiles that match neurophysiology in EC. What's more, there is also similarity in terms of when these distinct neuron types emerge during training/development. The EC has long been thought to be involved in path integration and localization of the animal's location. The general agreement between the different response properties in our model and the neurophysiology provide strong evidence supporting the hypothesis that the neural population in EC may provide an efficient code for representation self-locations based on the velocity input. Recently, there has been increased interest in using complex neural network models to understand the neural code. But the focus has been on using feedforward architectures, in particular CNNs BID29. Given the abundant recurrent connections in the brain, it seems a particularly fruitful avenue to take advantage of the recent development in RNNs to help with neuroscience questions BID34 BID50 BID37 BID53. Here, we only show one instance following this approach. However, the insight from this work could be general, and potentially useful for other cognitive functions as well. The finding that metabolic constraints lead to the emergence of grid-like responses may be seen as conceptually related to the efficient coding hypothesis in visual processing BID1, in particular the seminal work on the emergence of the V1-like Gabor filters in a sparse coding model by BID42. Indeed, our work is partly inspired by these . While there are conceptual similarities, however, we should also note there are differences between the sparse coding work and ours. First, the sparsity constraint in sparse coding can be naturally viewed as a particular prior while in the context of the recurrent network, it is difficult to interpret that way. Second, the grid-like responses are not the most sparse solution one could imagine. In fact, they are still quite dense compared to a more spatially localized representation. Third, the grid-like patterns that emerged in our network are not filters based on the raw input, rather the velocity inputs need to be integrated first in order to encode spatial locations. Our work is also inspired by recent work using the efficient coding idea to explain the functional architecture of the grid cells BID55. It has been shown that efficient coding considerations could explain the particular set of grid scales observed in rodents BID52. However, in that work, the firing patterns of the neurons are assumed to have a lattice structure to start with. Furthermore, our work is related to the study by Sussillo and others BID53, in which they show that regularization of RNN models are important for generating solutions that are similar to the neural activity observed in motor cortex. In Sussillo et al., a smoothness constraint together with others lead to simple oscillatory neural dynamics that well matches the neural data. We have not incorporated a smoothness constraint into our network. Additionally, we note that there are a few recent studies which use place cells as the input to generate grid cells BID8 BID51, which are fundamentally different from our work. In these feedforward network models, the grid cells essentially perform dimensionality reduction based on the spatial input from place cells. However, the main issue with these models is that, it is unclear how place cells acquire spatial tuning in the first place. To the contrary, our model takes the animal's velocity as the input, and addresses the question of how the spatial tuning can be generated from such input, which are known to exist in EC BID46 BID26. In another related study BID21, the authors train a RNN with LSTM units BID18 to perform different navigation tasks. However, no grid-like spatial firing patterns are reported. Although our model shows a qualitative match to the neural responses observed in the EC, nonetheless it has several major limitations, with each offering interesting future research directions. First, the learning rule we use seems to be biologically implausible. We are interested in exploring how a more biologically plausible learning rule could give rise to similar BID32 BID37 BID14. Second, the simulation do not show a variety of spatial scales in grid-like cells. Experimentally, it is known that grid cells have multiple spatial scales, that scale geometrically with a ratio 1.4 BID52, and this particular scale ratio is predicted by efficient coding of space BID55. We are investigating how to modify the model to get a hierarchy of spatial scales, perhaps by incorporating more neurons or modifying the regularization. Last but not least, we have focused on the representation produced by the trained RNN. An equally important set of questions concern how the networks actually support the generation of such a representation. As a preliminary effort, we have examined the connectivity patterns of the trained network, and they do not seem to resemble the connectivity patterns required by standard attractor network models. Maybe this should not be seen as too surprising. After all, the trained networks can produce a diverse set of neural responses, while the previous models only led to grid responses. It would be interesting for future work to systematically examine the questions related to the underlying mechanisms. To quantify the speed selectivity of each unit we first fit a line to the tuning curve of unit activity as a function of speed. The speed selectivity is the absolute value of the slope. If the unit activity is not modulated by speed then the speed selectivity is 0. To quantify the direction selectivity of each unit we calculated the average unit activity as a function of direction input and then took the maximum minus minimum of this tuning curve. If the unit activity is not modulated by direction then the direction selectivity is 0. To quantify the spatial selectivity we used lifetime sparseness BID56. If the unit activity is not modulated by spatial location then the spatial selectivity is 0. Each dot in the figures below show the selectivity for a single unit. Direc$on selec$vity"

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems