Question:
represent the scientific article to find a short tldr

E.g. 'A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multi-agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment. Many real-world scenarios that require cooperation among multiple autonomous agents are multi-goal multi-agent control problems: each agent needs to achieve its own individual goal, but the global optimum where all agents succeed is only attained when agents cooperate to allow the success of other agents. In autonomous driving, multiple vehicles must execute cooperative maneuvers when their individual goal locations and nominal trajectories are in conflict (e.g., double lane merges) . In social dilemmas, mutual cooperation has higher global payoff but agents' individual goals may lead to defection out of fear or greed . Even settings with a global objective that seem unfactorizable can be formulated as multi-goal problems: in Starcraft II micromanagement, a unit that gathers resources must not accidentally jeopardize a teammate's attempt to scout the opponent base ; in traffic flow optimization, different intersection controllers may have local throughput goals but must cooperate for high global performance . While the framework of multi-agent reinforcement learning (MARL) (; ;) has been equipped with methods in deep reinforcement learning (RL) and shown promise on high-dimensional problems with complex agent interactions (; ; ; ;), learning multi-agent cooperation in the multi-goal scenario involves significant open challenges. First, given that exploration is crucial for RL and even more so in MARL with larger state and joint action spaces, how should agents explore to learn both individual goal attainment and cooperation for others' success? Uniform random exploration is common in deep MARL but can be highly inefficient as the value of cooperative actions may be discoverable only in small regions of state space where cooperation is needed. Furthermore, the conceptual difference between attaining one's own goal and cooperating for others' success calls for more modularized and targeted approaches. Second, while there are methods for multi-agent credit assignment when all agents share a single goal (i.e., a global reward) (; ;), and while one could treat the cooperative multi-goal scenario as a problem with a single joint goal, this coarse approach makes it extremely difficult to evaluate the impact of an agent's action on another agent's success. Instead, the multi-goal scenario can benefit from fine-grained credit assignment that leverages available structure in action-goal interactions, such as local interactions where only few agents affect another agent's goal attainment at any time. Given these open challenges, our paper focuses on the cooperative multi-goal multi-agent setting where each agent is assigned a goal 1 and must learn to cooperate with other agents with possibly different goals. To tackle the problems of efficient exploration and credit assignment in this complex problem setting, we develop CM3, a novel general framework involving three synergistic components: 1. We approach the difficulty of multi-agent exploration from a novel curriculum learning perspective, by first training an actor-critic pair to achieve different goals in an induced single-agent setting (Stage 1), then using them to initialize all agents in the multi-agent environment (Stage 2). The key insight is that agents who can already act toward individual objectives are better prepared for discovery of cooperative solutions with additional exploration once other agents are introduced. In contrast to hierarchical learning where sub-goals are selected sequentially in time , all agents act toward their goals simultaneously in Stage 2 of our curriculum. 2. Observing that a wide array of complex MARL problems permit a decomposition of agents' observations and state vectors into components of self, others, and non-agent specific environment information , we employ function augmentation to bridge Stages 1-2: we reduce the number of trainable parameters of the actor-critic in Stage 1 by limiting their input space to the part that is sufficient for single-agent training, then augment the architecture in Stage 2 with additional inputs and trainable parameters for learning in the multi-agent environment. 3. We propose a credit function, which is an action-value function that specifically evaluates actiongoal pairs, for localized credit assignment in multi-goal MARL. We use it to derive a multi-goal multi-agent policy gradient for Stage 2. In synergy with the curriculum, the credit function is constructed via function augmentation from the critic in Stage 1. We evaluate our method on challenging multi-goal multi-agent environments with high-dimensional state spaces: cooperative navigation with difficult formations, double lane merges in the SUMO simulator , and strategic teamwork in a Checkers game. CM3 solved all domains significantly faster than IAC and COMA , and solved four out of five environments significantly faster than QMIX . Exhaustive ablation experiments show that the combination of all three components is crucial for CM3's overall high performance. While early theoretical work analyzed Markov games in discrete state and action spaces (; ;), recent literature have leveraged techniques from deep RL to develop general algorithms for high dimensional environments with complex agent interactions (; ;), which pose difficulty for traditional methods that do not generalize by learning interactions . Cooperative multi-agent learning is important since many real-world problems can be formulated as distributed systems in which decentralized agents must coordinate to achieve shared objectives . The multi-agent credit assignment problem arises when agents share a global reward . While credit assignment be resolved when independent individual rewards are available , this may not be suitable for the fully cooperative setting: showed that agents whose rewards depend on the success of other agents can cooperate better than agents who optimize for their own success. In the special case when all agents have a single goal and share a global reward, COMA et al., 2019) apply to agents with different rewards, they do not address multi-goal cooperation as they do not distinguish between cooperation and competition, despite the fundamental difference. Multi-goal MARL was considered in , who analyzed convergence in a special networked setting restricted to fully-decentralized training, while we conduct centralized training with decentralized execution . In contrast to multi-task MARL, which aims for generalization among non-simultaneous tasks , and in contrast to hierarchical methods that sequentially select subtasks , our decentralized agents must cooperate concurrently to attain all goals. Methods for optimizing high-level agent-task assignment policies in a hierarchical framework are complementary to our work, as we focus on learning low-level cooperation after goals are assigned. Prior application of curriculum learning to MARL include a single cooperative task defined by the number of agents and the probability of agent appearance , without explicit individual goals. instantiate new neural network columns for task transfer in single-agent RL. Techniques in transfer learning are complementary to our novel curriculum approach to MARL. In multi-goal MARL, each agent should achieve a goal drawn from a finite set, cooperate with other agents for collective success, and act independently with limited local observations. We formalize the problem as an episodic multi-goal Markov game, review an actor-critic approach to centralized training of decentralized policies, and summarize counterfactual-based multi-agent credit assignment. Multi-goal Markov games. A multi-goal Markov game is a tuple S, {O n}, {A n}, P, R, G, N, γ with N agents labeled by n ∈ [N]. In each episode, each agent n has one fixed goal g n ∈ G that is known only to itself. At time t and global state s t ∈ S, each agent n receives an observation o n t:= o n (s t) ∈ O n and chooses an action a n t ∈ A n. The environment moves to s t+1 due to joint action a t:= {a 1 t, . . ., a N t}, according to transition probability P (s t+1 |s t, a t). Each agent receives a reward R n t:= R(s t, a t, g n), and the learning task is to find stochastic decentralized policies π n: O n × G × A n →, conditioned only on local observations and goals, to maximize where γ ∈ and joint policy π factorizes as π(a|s, g):= N n=1 π n (a n |o n, g n) due to decentralization. Let a −n and g −n denote all agents' actions and goals, respectively, except that of agent n. Let boldface a and g denote the joint action and joint goals, respectively. For brevity, let π(a n):= π n (a n |o n, g n). This model covers a diverse set of cooperation problems in the literature , without constraining how the attainability of a goal depends on other agents: at a traffic intersection, each vehicle can easily reach its target location if not for the presence of other vehicles; in contrast, agents in a strategic game may not be able to maximize their rewards in the absence of cooperators . Centralized learning of decentralized policies. A centralized critic that receives full state-action information can speed up training of decentralized actors that receive only local information . Directly extending the single-goal case, for each n ∈ [1..N] in a multigoal Markov game, critics are represented by the value function V π n (s):= E π ∞ t=0 γ t R n t s 0 = s and the action-value function Q π n (s, a):= E π ∞ t=0 γ t R n t s 0 = s, a 0 = a, which evaluate the joint policy π against the reward R n for each goal g n. Multi-agent credit assignment. In MARL with a single team objective, COMA addresses credit assignment by using a counterfactual baseline in an advantage function (, Lemma 1), which evaluates the contribution of a chosen action a n versus the average of all possible counterfactualsâ n, keeping a −n fixed. The analysis in for a formally equivalent action-dependent baseline in RL suggests that COMA is a low-variance estimator for single-goal MARL. We derive its variance in Appendix C.1. However, COMA is unsuitable for credit assignment in multi-goal MARL, as it would treat the collection of goals g as a global goal and only learn from total reward, making it extremely difficult to disentangle each agent's impact on other agents' goal attainment. Furthermore, a global Q-function does not explicitly capture structure in agents' interactions, such as local interactions involving a limited number of agents. We substantiate these arguments by experimental in Section 6. We describe the complete CM3 learning framework as follows. First we define a credit function as a mechanism for credit assignment in multi-goal MARL, then derive a new cooperative multi-goal policy gradient with localized credit assignment. Next we motivate the possibility of significant training speedup via a curriculum for multi-goal MARL. We describe function augmentation as a mechanism for efficiently bridging policy and value functions across the curriculum stages, and finally synthesize all three components into a synergistic learning framework. If all agents take greedy goal-directed actions that are individually optimal in the absence of other agents, the joint action can be sub-optimal (e.g. straight-line trajectory towards target in traffic). Instead rewarding agents for both individual and collective success can avoid such bad local optima. A naïve approach based on previous works would evaluate the joint action a via a global Q-function Q π n (s, a) for each agent's goal g n, but this does not precisely capture each agent's contribution to another agent's attainment of its goal. Instead, we propose an explicit mechanism for credit assignment by learning an additional function Q π n (s, a m) that evaluates pairs of action a m and goal g n, for use in a multi-goal actor-critic algorithm. We define this function and show that it satisfies the classical relation needed for sample-based model-free learning. Definition 1. For n, m ∈ [N], s ∈ S, the credit function for goal g n and a m ∈ A m by agent m is:, the credit function satisfies the following relations: Derivations are given in Appendix B.1, including the relation between Q π n (s, a m) and Q π n (s, a). Equation takes the form of the Bellman expectation equation, which justifies learning the credit function, parameterized by θ Qc, by optimizing the standard loss function in deep RL: While centralized training means the input space scales linearly with agent count, many practical environments involving only local interactions between agents allows centralized training with few agents while retaining decentralized performance when deployed at scale (evidenced in Appendix E). We use the credit function as a critic within a policy gradient for multi-goal MARL. Letting θ parameterize π, the overall objective J(π) is maximized by ascending the following gradient: Proposition 2. The cooperative multi-goal credit function based MARL policy gradient is This is derived in Appendix B.2. For a fixed agent m, the inner summation over n considers all agents' goals g n and updates m's policy based on the advantage of a m over all counterfactual actionŝ a m, as measured by the credit function for g n. The strength of interaction between action-goal pairs is captured by the extent to which Q π n (s,â m) varies withâ m, which directly impacts the magnitude of the gradient on agent m's policy. For example, strong interaction in non-constant Q π n (s, ·), which implies larger magnitude of A π n,m and larger weight on ∇ θ log π(a m). The double summation accounts for first-order interaction between all action-goal pairs, but complexity can be reduced by omitting terms when interactions are known to be sparse, and our empirical runtimes are on par with other methods due to efficient batch computation (Appendix F). As the second term in A π n,m is a baseline, the reduction of variance can be analyzed similarly to that for COMA, given in Appendix C.2.), ablation show stability improvement due to the credit function (Section 6). As the credit function takes in a single agent's action, it synergizes with both CM3's curriculum and function augmentation as described in Section 4.5. Multi-goal MARL poses a significant challenge for exploration. Random exploration can be highly inefficient for concurrently learning both individual task completion and cooperative behavior. Agents who cannot make progress toward individual goals may rarely encounter the region of state space where cooperation is needed, rendering any exploration useless for learning cooperative behavior. On the other extreme, exploratory actions taken in situations that require precise coordination can easily lead to penalties that cause agents to avoid the coordination problem and fail to achieve individual goals. Instead, we hypothesize and confirm in experiments that agents who can achieve individual goals in the absence of other agents can more reliably produce state configurations where cooperative solutions are easily discovered with additional exploration in the multi-agent environment 2. We propose a MARL curriculum that first solves a single-agent Markov decision process (MDP), as preparation for subsequent exploration speedup. Given a cooperative multi-goal Markov game MG, we induce an MDP M to be the tuple S n, O n, A n, P n, R, γ, where an agent n is selected to be the single agent in M. Entities S n, P n, and R are defined by removing all dependencies on agent interactions, so that only components depending on agent n remain. This reduction to M is possible in almost all fully cooperative multi-agent environments used in a large body of work 3 , precisely because they support a variable number of agents, including N = 1. Important real-world settings that allow this reduction include autonomous driving, multi traffic light control, and warehouse commissioning (removing all but one car/controller/robot, respectively, from the environment). Given a full Markov game implementation, the reduction involves only deletion of components associated with all other agents from state vectors (since an agent is uniquely defined by its attributes), deletion of if-else conditions from the reward function corresponding to agent interactions, and likewise from the transition function if a simulation is used. Appendix G provides practical guidelines for the reduction. Based on M, we define a greedy policy for MG. Definition 2. A greedy policy π n by agent n for cooperative multi-goal MG is defined as the optimal policy π * for the induced MDP M where only agent n is present. This naturally leads to our proposed curriculum: Stage 1 trains a single agent in M to achieve a greedy policy, which is then used for initialization in MG in Stage 2. Next we explain in detail how to leverage the structure of decentralized MARL to bridge the two curriculum stages. In Markov games with decentralized execution, an agent's observation space decomposes into self captures the agent's own properties, which must be observable by the agent for closed-loop control, while o n others ∈ O n others is the agent's egocentric observation of other agents. In our work, egocentric observations are private and not accessible by other agents . Similarly, global state s decomposes into s:= (s env, s n, s −n), where Function augmentation π π 1 Figure 1: In Stage 1, Q 1 and π 1 learn to achieve multiple goals in a single-agent environment. Between Stage 1 and 2, π is constructed from the trained π 1 and a new module π 2 according to (same construction is done for Q n (s, a) and Q n (s, a m), not shown). In the multi-agent environment of Stage 2, these augmented functions are instantiated for each of N agents (with parameter-sharing). s env is environment information not specific to any agent (e.g., position of a landmark), and s n captures agent n's information. While this decomposition is implicitly available in a wide range of complex multi-agent environments (; ; ; ; ;), we explicitly use it to implement our curriculum. In Stage 1, as the ability to process o n others and s −n is unnecessary, we reduce the input space of policy and value functions, thereby reducing the number of trainable parameters and lowering the computation cost. In Stage 2, we restore Stage 1 parameters and activate new modules to process additional inputs o n others and s −n. This augmentation is especially suitable for efficiently learning the credit function and global Q-function, since Q(s, a) can be augmented into both Q π n (s, a) and Q π n (s, a m), as explained below. We combine the preceding components to create CM3, using deep neural networks for function approximation (Figure 1 and Algorithm 1). Without loss of generality, we assume parameter-sharing among homogeneous agents with goals as input . The inhomogeneous case can be addressed by N actor-critics. Drawing from multi-task learning , we sample goal(s) in each episode for the agent(s), to train one model for all goals. Stage 1. We train an actor π 1 (a|o, g) and critic Q 1 (s 1, a, g) to convergence according to and Stage 2. The Markov game is instantiated with all N agents. We restore the trained π 1 parameters, instantiate a second neural network π 2 for agents to process o n others, and connect the output of π 2 to a selected hidden layer of π 1. Concretely, let h Being restored from Stage 1, not re-initialized, hidden layers i < i * begin with the ability to process (o n self, g n), while the new weights in π 2 and W 1:2 specifically learn the effect of surrounding agents. Higher layers i ≥ i * that already take greedy actions to achieve goals in Stage 1 must now do so while cooperating to allow other agents' success. This augmentation scheme is simplest for deep policy and value networks using fully-connected or convolutional layers. The middle panel of Figure 1 depicts the construction of π from π 1 and π 2. The global Q π (s, a, g n) is constructed from Q 1 similarly: when the input to Q 1 is (s env, s n, a n, g n), a new module takes input (s −n, a −n) and connects to a chosen hidden layer of 4 Setting i * to be the last hidden layer worked well in our experiments, without needing to tune. augmented from a copy of Q 1, such that when We train the policy using, train the credit function with loss, and train the global Q-function with the joint-action analogue of. We investigated the performance and robustness of CM3 versus existing methods on diverse and challenging multi-goal MARL environments: cooperative navigation in difficult formations, double lane merge in autonomous driving, and strategic cooperation in a Checkers game. We evaluated ablations of CM3 on all domains. We describe key setup here, with full details in Appendices G to J. Cooperative navigation: We created three variants of the cooperative navigation scenario in , where N agents cooperate to reach a set of targets. We increased the difficulty by giving each agent only an individual reward based on distance to its designated target, not a global team reward, but initial and target positions require complex cooperative maneuvers to avoid collision penalties (Figure 3). Agents observe relative positions and velocities (details in Appendix G.1). SUMO: Previous work modeled autonomous driving tasks as MDPs in which all other vehicles do not learn to respond to a single learning agent . However, real-world driving requires cooperation among different drivers' with personal goals. Built in the SUMO traffic simulator with sublane resolution , this experiment requires agent vehicles to learn double-merge maneuvers to reach goal lane assignments (Figure 4). Agents have limited field of view and receive sparse rewards (Appendix G.2). Checkers: We implemented a challenging strategic game (Appendix G.3, an extension of), to investigate whether CM3 is beneficial even when an agent cannot maximize its reward in the absence of another agent. In a gridworld with red and yellow squares that disappear when collected (Figure 2), Agent A receives +1 for red and -0.5 for yellow; Agent B receives -0.5 for red and +1 for yellow. Both have a limited 5x5 field of view. The global optimum requires each agent to clear the path for the other. Algorithm implementations. We describe key points here, leaving complete architecture details and hyperparameter tables to Appendices H and I. CM3: Stage 1 is defined for each environment as follows (Appendix G): in cooperative navigation, a single particle learns to reach any specified landmark; in SUMO, a car learns to reach any specified goal lane; in Checkers, we alternate between training one agent as A and B. Appendix H describes function augmentation in Stage 2 of CM3. COMA : the joint goal g and total reward n R n can be used to train COMA's global Q function, which receives input (s, o n, g n, n, a −n, g −n). Each output node i represents Q(s, a n = i, a −n, g). IAC : IAC trains each agent's actor and critic independently, using the agent's own observation. The TD error of value function V (o n, g n) is used in a standard policy gradient . QMIX : we used the original hypernetwork, giving all goals to the mixer and individual goals to each agent network. We used a manual coordinate descent on exploration and learning rate hyperparameters, including values reported in the original works. We ensured the number of trainable parameters are similar among all methods, up to method-specific architecture requirements for COMA and QMIX. Ablations. We conducted ablation experiments in all domains. To discover the speedup from the curriculum with function augmentation, we trained the full Stage 2 architecture of CM3 (labeled as Direct) without first training components π 1 and Q 1 in an induced MDP. To investigate the benefit of the new credit function and multi-goal policy gradient, we trained an ablation (labeled QV) with, where credit assignment between action-goal pairs is lost. QV uses the same π 1, Q 1, and function augmentation as CM3. CM3 finds optimal or near-optimal policies significantly faster than IAC and COMA on all domains, and performs significantly higher than QMIX in four out of five. We report absolute runtime in Appendix F and account for CM3's Stage 1 episodes (Appendix J) when comparing sample efficiency. Main comparison. Over all cooperative navigation scenarios (Figures 5a to 5c), CM3 (with 1k episodes in Stage 1) converged more than 15k episodes faster than IAC. IAC reached the same final performance as CM3 because dense individual rewards simplifies the learning problem for IAC's fully decentralized approach, but CM3 benefited significantly from curriculum learning, as evidenced by comparison to "Direct" in Figure 5f. QMIX and COMA settled at suboptimal behavior. Both learn global critics that use all goals as input, in contrast to CM3 and IAC that process each goal separately. This indicates the difficulty of training agents for individual goals under a purely global approach. While COMA was shown to outperform IAC in SC2 micromanagement where IAC must learn from a single team reward , our IAC agents have access to individual rewards that resolve the credit assignment issue and improve performance . In SUMO (Figure 5d), CM3 and QMIX found cooperative solutions with performances within the margin of error, while COMA and IAC could not break out of local optima where vehicles move straight but do not perform merge maneuvers. Since initial states force agents into the region of state space requiring cooperation, credit assignment rather than exploration is the dominant challenge, which CM3 addressed via the credit function, as evidenced in Figure 5i. IAC underperformed because SUMO requires a longer sequence of cooperative actions and gave much sparser rewards than the "Merge" scenario in cooperative navigation. We also show that centralized training of merely two decentralized agents allows them to generalize to settings with much heavier traffic (Appendix E). In Checkers (Figure 5e), CM3 (with 5k episodes in Stage 1) converged 10k episodes faster than COMA and QMIX to the global optimum with score 24. Both exploration of the combinatorially large joint trajectory space and credit assignment for path clearing are challenges that CM3 successfully addressed. COMA only solved Checkers among all domains, possibly because the small bounded environment alleviates COMA's difficulty with individual goals in large state spaces. IAC underperformed all centralized learning methods because cooperative actions that give no instantaneous reward are hard for selfish agents to discover in Checkers. These demonstrate CM3's ability to attain individual goals and find cooperative solutions in diverse multi-agent systems. Ablations. The significantly better performance of CM3 versus "Direct" (Figures 5f to 5j) shows that learning individual goal attainment prior to learning multi-agent cooperation, and initializing Stage 2 with Stage 1 parameters, are crucial for improving learning speed and stability. It gives evidence that while global action-value and credit functions may be difficult to train from scratch, function augmentation significantly eases the learning problem. While "QV" initially learns quickly to attain individual goals, it does so at the cost of frequent collisions, higher variance, and inability to maintain a cooperative solution, giving clear evidence for the necessity of the credit function. We presented CM3, a general framework for cooperative multi-goal MARL. CM3 addresses the need for efficient exploration to learn both individual goal attainment and cooperation, via a two-stage curriculum bridged by function augmentation. It achieves local credit assignment between action and goals using a credit function in a multi-goal policy gradient. In diverse experimental domains, CM3 attains significantly higher performance, faster learning, and overall robustness than existing MARL methods, displaying strengths of both independent learning and centralized credit assignment while avoiding shortcomings of existing methods. Ablations demonstrate each component is crucial to the whole framework. Our motivate future work on analyzing CM3's theoretical properties and generalizing to inhomogeneous systems or settings without known goal assignments. Hernandez-Leal, P., Kartal, B., and Taylor Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R. Instantiate N > 1 agents 8: Set all target network weights to equal main networks weights 13: Initialize exploration parameter = start and empty replay buffer B for each training episode e = 1 to E do for t = 1 to T do // execute policies in environment Sample action a n t ∼ π(a n t |o n t ; θ π,) for each agent. Compute global target for all n: Gradient descent on L(θ Qg) = if c = 1 then 29: end if 35: Update policy: Update all target network parameters using: Reset buffer B Off-policy training with a large replay buffer allows RL algorithms to benefit from less correlated transitions . The algorithmic modification for off-policy training is to maintain a circular replay buffer that does not reset (i.e. remove line 38), and conduct training (lines 24-41) while executing policies in the environment (lines 17-22). Despite introducing bias in MARL, we found that off-policy training benefited CM3 in SUMO and Checkers. By stationarity and relabeling t, the credit function can be written: Using the law of iterated expectation, the credit function satisfies the Bellman expectation equation: The goal-specific joint value function is the marginal of the credit function: The credit function can be expressed in terms of the goal-specific action-value function: First we state some elementary relations between global functions V π n (s) and Q π n (s, a). These carry over directly from the case of an MDP, by treating the joint policy π as as an effective "single-agent" policy and restricting attention to a single goal g n (standard derivations are included at the end of this section). We follow the proof of the policy gradient theorem : We can replace Q π n (s, a) by the advantage function A π n (s, a):= Q π n (s, a) − V π n (s), which does not change the expectation in Equation because: So the gradient can be written Recall that from, for any choice of agent label k ∈ [1..N]: Then substituting into: Now notice that the choice of k in is completely arbitrary, since holds for any k ∈ [1..N]. Therefore, it is valid to distribute A π n,k (s, a) into the summation in using the summation index m instead of k. Further summing over all n, we arrive at the of Proposition 2: The relation between V π n (s) and Q π n (s, a) in and are derived as follows: Let Q:= Q π (s, a, g) denote the centralized Q function, let π(a n):= π(a n |o n, g n) denote a single agent's policy, and let π(a −n):= π(a −n |o −n, g −n) denote the other agents' joint policy. In cooperative multi-goal MARL, the direct application of COMA has the following gradient. Define the following: f n ]. its variance can be derived to be : The CM3 gradient can be rewritten as As before, z m:= ∇ θ log π(a m). Define h nm:= z m (Q n − b nm (s)) and let h n:= m h nm. Then the variance is A greedy initialization can provide significant improvement in multi-agent exploration versus naïve random exploration, as shown by a simple thought experiment. Consider a two-player MG defined by a 4 × 3 gridworld with unit actions (up, down, left, right). Agent A starts at with goal, while agent B starts at with goal. The greedy policy for each agent in MG is to move horizontally toward its target, since this is optimal in the induced M (when the other agent is absent). Case 1: Suppose that for ∈, A and B follow greedy policies with probability 1 −, and take random actions (p(a) = 1/4) with probability. Then the probability of a symmetric optimal trajectory is P (cooperate) = 2 2 ((1 −) + /4) 8. For = 0.5, P (cooperate) ≈ 0.01. Case 2: If agents execute uniform random exploration, then P (cooperate) = 3.05e-5 0.01. We investigated whether policies trained with few agent vehicles (N = 2) on an empty road can generalize to situations with heavy SUMO-controlled traffic. We also tested on initial and goal lane configurations (C3 and C4) which occur with low probability when training with configurations C1 and C2. Table 1 shows the sum of agents' reward, averaged over 100 test episodes, on these configurations that require cooperation with each other and with minimally-interactive SUMOcontrolled vehicles for success. CM3's higher performance than IAC and COMA in training is reflected by better generalization performance on these test configurations. There is almost negligible decrase in performance from train Figure 5d to test, giving evidence to our hypothesis that centralized training with few agents is feasible even for deployment in situations with many agents, for certain applications where local interactions are dominant. F ABSOLUTE RUNTIME CM3's higher sample efficiency does not come at greater computational cost, as all methods' runtimes are within an order of magnitude of one another. Test times have no significant difference as all neural networks were similar. The full Markov game for each experimental domain, along with the single-agent MDP induced from the Markov game, are defined in this section. In all domains, each agent's observation in the Markov game consists of two components, o self and o others. CM3 leverages this decomposition for faster training, while IAC, COMA and QMIX do not. This domain is adapted from the multi-agent particle environment in. Movable agents and static landmarks are represented as circular objects located in a 2D unbounded world with real-valued position and velocity. Agents experience contact forces during collisions. A simple model of inertia and friction is involved. State. The global state vector is the concatenation of all agents' absolute position (x, y) ∈ R 2 and velocity (v x, v y) ∈ R 2. Observation. Each agent's observation of itself, o self, is its own absolute position and velocity. Each agent's observation of others, o others, is the concatenation of the relative positions and velocities of all other agents with respect to itself. Actions. Agents take actions from the discrete set do nothing, up, down, left, right, where the movement actions produce an instantaneous velocity (with inertia effects). Goals and initial state assignment. With probability 0.2, landmarks are given uniform random locations in the set (−1, 1) 2, and agents are assigned initial positions uniformly at random within the set (−1, 1) 2. With probability 0.8, they are predefined as follows (see Figure 3). In "Antipodal", landmarks for agents 1 to 4 have (x, y) coordinates [(0.9,0.9), (-0.9,-0.9), (0.9,-0.9), (-0.9,0.9)], while agents 1 to 4 are placed at [(-0.9,-0.9), (0.9,0.9), (-0.9,0.9), (0.9,-0.9)]. In "Intersection", landmark coordinates are [(0.9,-0.15), (-0.9,0.15), (0.15,0.9), (-0.15,-0.9 Reward. At each time step, each agent's individual reward is the negative distance between its position and the position of its assigned landmark. If a collision occurs between any pair of agents, both agents receive an additional -1 penalty. A collision occurs when two agents' distance is less than the sum of their radius. Termination. Episode terminates when all agents are less than 0.05 distance from assigned landmarks. Induced MDP. This is the N = 1 case of the Markov game, used by Stage 1 of CM3. The single agent only receives o self . In each episode, its initial position and the assigned landmark's initial position are both uniform randomly chosen from (−1, 1) 2. We constructed a straight road of total length 200m and width 12.8m, consisting of four lanes. All lanes have width 3.2m, and vehicles can be aligned along any of four sub-lanes within a lane, with lateral spacing 0.8m. Vehicles are emitted at average speed 30m/s with small deviation. Simulation time resolution was 0.2s per step. SUMO file merge_stage3_dense.rou.xml contains all vehicle parameters, and merge.net.xml defines the complete road architecture. State. The global state vector s is the concatenation of all agents' absolute position (x, y), normalized respectively by the total length and width of the road, and horizontal speed v normalized by 29m/s. Observation. Each agent observation of itself o n self is a vector consisting of: agent speed normalized by 29m/s, normalized number of sub-lanes between agent's current sub-lane and center sub-lane of goal lane, and normalized longitudinal distance to goal position. Each agent's observation of others o n others is a discretized observation tensor of shape centered on the agent, with two channels: binary indicator of vehicle occupancy, and normalized relative speed between agent and other vehicles. Each channel is a matrix with shape, corresponding to visibility of 15m forward and backward (with resolution 2.5m) and four sub-lanes to the left and right. Actions. All agents have the same discrete action space, consisting of five options: no-op (maintain current speed and lane), accelerate (2.5m/s 2), decelerate (−2.5m/s 2), shift one sub-lane to the left, shift one sub-lane to the right. Each agent's action a n is represented as a one-hot vector of length 5. Goals and initial state assignment. Each goal vector g n is a one-hot vector of length 4, indicating the goal lane at which agent n should arrive once it crosses position x=190m. With probability 0.2, agents are assigned goals uniformly at random, and agents are assigned initial lanes uniformly at random at position x=0. With probability 0.8, agent 1's goal is lane 2 and agent 2's goal is lane 1, while agent 1 is initialized at lane 1 and agent 2 is initialized at lane 2 (see Figure 4). Departure times were drawn from a normal distribution with mean 0s and standard deviation 0.5s for each agent. Reward. The reward R(s t, a t, g n) for agent n with goal g n is given according to the conditions: -1 for a collision; -10 for time-out (exceed 33 simulation steps during an episode); 10(1 − ∆) for reaching the end of the road and having a normalized sub-lane difference of ∆ from the center of the goal lane; and -0.1 if current speed exceeds 35.7m/s. Termination. Episode terminates when 33 simulation steps have elapsed or all agents have x >190m. Induced MDP. This is the N = 1 case of the Markov game defined above, used by Stage 1 of CM3. The single agent receives only o self. For each episode, agent initial and goal lanes are assigned uniformly at random from the available lanes. This domain is adapted from the Checkers environment in. It is a gridworld with 5 rows and 13 columns (Figure 2). Agents cannot move to the two highest and lowest rows and the two highest and lowest columns, which are placed for agents' finite observation grid to be well-defined. Agents cannot be in the same grid location. Red and yellow collectible reward are placed in a checkered pattern in the middle 3x8 region, and they disappear when any agent moves to their location. State. The global state s consists of two components. The first is s T, a tensor of shape, where the two "channels" in the last dimension represents the presence/absence of red and yellow rewards as 1-hot matrices. The second is s V, the concatenation of all agents' (x, y) location (integer-valued) and the number of red and yellow each agent has collected so far. Observation. Each agent's obsevation of others, o n others, is the concatenation of all other agents' normalized coordinates (normalized by total size of grid). An agent's observation of itself, o n self, consists of two components. First, o n self,V is a vector concatenation of agent n's normalized coordinate and the number of red and yellow it has collected so far. Second, o n self,T is a tensor of shape, centered on its current location in the grid. The tensor has three "channels", where the first two represent presence/absence of red and yellow rewards as 1-hot matrices, and the last channel indicates the invalid locations as a 1-hot matrix. The agent's own grid location is a valid location, while other agents' locations are invalid. Actions. Agents choose from a discrete set of actions do-nothing, up, down, left, right. Movement actions transport the agent one grid cell in the chosen direction. Goals. Agent A's goal is to collect all red rewards without touching yellow. Agent B's goal is to collect all yellow without touching red. The goal is represented as a 1-hot vector of length 2. Reward. Agent A gets +1 for red, -0.5 for yellow. Agent B gets -0.5 for red, +1 for yellow. For all experiment domains, ReLU nonlinearity was used for all neural network layers unless otherwise specified. All layers are fully-connected feedforward layers, unless otherwise specified. All experiment domains have a discrete action space (with |A| = 5 actions), and action probabilities were computed by lower-bounding softmax outputs of all policy networks by P (a n = i) = (1 −)softmax(i) + /|A|, where is a decaying exploration parameter. To keep neural network architectures as similar as possible among all algorithms, our neural networks for COMA differ from those of in that we do not use recurrent networks, and we do not feed previous actions into the Q function. For the Q network in all implementations of COMA, the value of each output node i is interpreted as the action-value Q(s, a −n, a n = i, g) for agent n taking action i and all other agents taking action a −n. Also for COMA, agent n's label vector (one-hot indicator vector) and observation o self were used as input to COMA's global Q function, to differentiate between evaluations of the Q-function for different agents. These were choices in COMA. COMA uses the same policy network as Stage 2 of CM3. The global Q function of COMA computes Q(s, (a n, a −n)) for each agent n as follows. Input is the concatenation of state s, all other agents' 1-hot actions a −n, agent n's goal g n, all other agent goals g −n, agent label n, and agent n's observation o n self. This is passed through two layers of 128 units each, then connected to a linear output layer with 5 units. The Q 1 function in Stage 1 feeds the concatenation of state s, goal g, and 1-hot action a to one layer with 256 units, which is connected to the special layer h The Q 1 function in Stage 1 is defined as: state tensor s T is fed to a convolutional layer with 4 filters of size 3x5 and stride 1x1 and flattened. o n self,T is given to a convolution layer with 6 filters of size 3x3 and stride 1x1 and flattened. Both are concatenated with s n (agent n part of the s V vector), goal g n, action a n and o COMA. COMA uses the same policy network as Stage 2 of CM3. The global Q(s, (a n, a −n)) function of COMA is defined as follows for each agent n. Tensor part of global state s T is given to a convolutional layer with 4 filters of size 3x5 and stride 1x1. Tensor part of agent n's observation o n self,T is given to a convolutional layer with 6 filters of size 3x3 and stride 1x1. Outputs of both convolutional layers are flattened, then concatenated with s V, all other agents' actions a −n, agent n's goal g n, other agents' goals g −n, agent n's label vector, and agent n's vector observation o n self,V. The concatenation is passed through two layers with 256 units each, then to a linear output layer with 5 units. QMIX. Individual value functions are defined as: o n self,T is passed through the same convolutional layer as above, connected to hidden layer with 32 units, then concatenated with o n self,V, a n t−1, and g n. This is connected to layer h 2 with 64 units. o n others is connected to a layer with 64 units then connectd to h 2. h 2 is fully-connected to an output layer. The mixing network feeds s T into the same convolutional network as above and follows the exact architecture of with embedding dimension 128. We used the Adam optimizer in Tensorflow with hyperparameters in Tables 3 to 5. div is used to compute the exploration decrement step:= (start − end)/ div. 5e2 1e3 1e3 1e4 2e4 1e4 1e4 Replay buffer 1e4 1e4 1e4 1e4 1e4 1e4 1e4 Minibatch size 128 128 128 128 128 128 128 Steps per Stage 1 training curves for all three experimental domains are shown in Figure 6.' == 'A modular method for fully cooperative multi-goal multi-agent reinforcement learning, based on curriculum learning for efficient exploration and credit assignment for action-goal interactions.' != 'We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.'

Question: Deep Reinforcement Learning algorithms lead to agents that can solve difficult decision making problems in complex environments. However, many difficult multi-agent competitive games, especially real-time strategy games are still considered beyond the capability of current deep reinforcement learning algorithms, although there has been a recent effort to change this \citep{openai_2017_dota, vinyals_2017_starcraft}. Moreover, when the opponents in a competitive game are suboptimal, the current \textit{Nash Equilibrium} seeking, self-play algorithms are often unable to generalize their strategies to opponents that play strategies vastly different from their own. This suggests that a learning algorithm that is beyond conventional self-play is necessary. We develop Hierarchical Agent with Self-play (HASP), a learning approach for obtaining hierarchically structured policies that can achieve higher performance than conventional self-play on competitive games through the use of a diverse pool of sub-policies we get from Counter Self-Play (CSP). We demonstrate that the ensemble policy generated by HASP can achieve better performance while facing unseen opponents that use sub-optimal policies. On a motivating iterated Rock-Paper-Scissor game and a partially observable real-time strategic game (http://generals.io/), we are led to the that HASP can perform better than conventional self-play as well as achieve 77% win rate against FloBot, an open-source agent which has ranked at position number 2 on the online leaderboards. Deep reinforcement learning (RL) has achieved significant success on many complex sequential decision-making problems. Most of the problems are in robotics domain or video games BID16. However, complex real-time strategic competitive games still pose a strong challenge to the current deep reinforcement learning method due to the requirement of the ability to handle long-term scheduling, partial observability and multi-agent collaboration/competition. BID28 BID20. Competitive games such as Go, in which each player optimize their own interests by finding the best response to opponents' strategies, are usually studied mainly on finding the Nash Equilibrium solutions BID20, namely a combination of players' strategies upon which neither player can obtain higher rewards by modifying their strategy unilaterally BID23. However, in the real-world, opponents can have a variety of strengths and play styles and do not always adopt the equilibrium solutions. In fact, human players are often remarkably good at analyzing strategies, tendencies, and flaws in opponents' behavior and then exploiting the opponents even if the ing exploiting strategies themselves are subject to exploitation. Exploitation is a central component of sports and competitive games. This is also applicable in other real-world competitive domains, including airport and network security, financial and energy trading, traffic control, routing, etc. Therefore, exploring game-playing strategies that intentionally avoid the equilibrium solution and instead "learn to exploit" is a promising research direction toward more capable, adaptable, and ultimately more human-like artificial agents. Hence, we develop a new algorithm Hierarchical Agent with Self-Play that learns to exploit the suboptimality of opponents in order to learn a wider variety of behaviors more in line with what humans might choose to display. In this work, we focus on two-player, symmetric, extensive form games of imperfect information, though generalization to more players and asymmetric games is feasible and relatively straightforward. First, we adopt recent Proximal Policy Gradient (PPO) methods in deep reinforcement learning (RL), which has been successful at handling complex games BID16 BID24 BID17 and many other fields BID20 ) Second, we aim to automatically acquire a strong strategy that generalizes against opponents that we have not seen in training. Here we use self-play to gradually acquire more and more complex behaviors. This technique has proven successful at solving backgammon BID27, the game of Go, imperfect information games such as Poker BID8 BID9, continuous control BID0, and modern video games BID20.In this paper, we investigate a new method for learning strong policies on multi-player games. We introduce Hierarchical Agent with Self-Play, our hierarchical learning algorithm that automatically learns several diverse, exploitable polices and combines them into an ensemble model that draws on the experience of the sub-policies to respond appropriately to different opponents. Then, we show the of some experiments on two multiplayer games: iterated Rock-Paper-Scissors and a partially observable real-time strategy game based on a popular online game generals.io (http://generals.io/). We show that compared to conventional self-play, our algorithm learns a more diverse set of strategies and obtains higher rewards against test opponents of different skill levels. Remarkably, it can achieve 77% win rate against the FloBot, the strongest open-sourced scripted bot on the generals.io online leaderboard. Many real world multi-agent problems can be described by Markov games BID12. A Markov game of N players at time step t has a full state s t and assigns each player i (1 ≤ i ≤ N) an observation o t,i. Then player i samples an action a t,i from its policy DISPLAYFORM0 where O i and A i are the observation and action spaces. Given all players' actions, the environment transits to a new state s t+1 and sends a reward r t,i to player i. The goal for each player i is to maximize expected total discounted rewards DISPLAYFORM1 A typical characterization of optimal strategies / policies is a Nash equilibrium, DISPLAYFORM2.., π N ) for any other π i. Finding equilibrium strategies can be very challenging, even for single-step (a.k.a. normal form) games BID23. Most studies focus on either fully cooperative (r t,i = r t,j ∀i, j) or two-player, zero-sum (N = 2, r t,1 = −r t,2) competitive cases. Methods for cooperative games include optimistic and hysteretic Q learning BID10 BID14 BID19, and recently centralized critic with decentralized actors BID6 BID13. BID7 also shows the ability to find exploiting strategies; however it is in under relatively simple games and can be potentially combined with our method to have better performance. BID21 and BID4 have comprehensive surveys on this topic. For two-player, zero-sum games, though small problems can be solved by linear programming, more complex (and symmetric) ones usually require self-play and some form of learning. Self-play, namely training the agent against itself, is a powerful technique to bootstrap from an initially random agent. Classic game-theoretic techniques often offer provable convergence to Nash equilibrium, and some also achieve success while combined with deep reinforcement learning. Such methods include fictitious play BID2 BID8, counterfactual regret minimization BID29 BID18 BID3, replicator dynamics BID26, double oracle BID15 BID9, and so on. The combination of deep learning, planning, and self-play led to the famous Go-playing agents AlphaGo and AlphaZero. Most recently, self-play achieved success on full five versus five Dota 2, beating a team of 99.95th percentile Dota players . Games are often popular environments for reinforcement learning research, and recently there has been a heavy interest BID20 BID28 in real-time strategy (RTS) games. These games prove difficult for current methods due to the importance of long-term decisions as well as large action spaces. We propose generals.io (Generals) as an interesting and economical research environment that has many of the same challenges as Dota 2 and Starcraft II (SC2) while being very fast to simulate and having a sizable community of players online to evaluate against. (a) One action in the generals game. An agent needs to select which grid to execute and then perform an action to a one of the 4 adjacent grid. After each action, one of the arm will be left in the original grid and all the army will be moved to the new grid.(b) The army aggregation scheme in the game. For each turn, the taken city and generals grids will generate one more army on the grid. After 50 turns, all the plain grid will increase 1 army. Generals games in FIG0 take place on a rectangular grid. There can be anywhere from two to eight players at a time, although in this work we only consider the case of two players. Each player gets their own color, as well as a "General", and they cannot see any tile that is not adjacent to a tile they own. Moves happen every 0.5 seconds, and consist of each player moving some army from one tile to an adjacent tile. Players can choose the grid to move and then move their army freely between tiles that they own, and if a player decides to move its army into a tile that is not their own color, the amount of army they are moving is subtracted from that tile and if the total is now negative, the player takes ownership. Fig. 2a demonstrates an example of how to move the army. There are also cities scattered throughout the map, which have a high cost to conquer. The army aggregation mechanism is described in Fig. 2b Every other turn, each player gets one army on their general as well as on every city they own. Every 50 turns, each player also gets an additional army on each tile they own. The goal of the game is to take ownership of the opponent's general. The fog-of-war in Generals means that agents have to actively learn to seek out information. Agents also need to learn how to prevent themselves from being revealed, as well as learn to defend against incoming attacks, know when to strike, manage its own resources by expanding its own territory and investing in cities. Players online show a variety of strategies, including rushing to their opponent in the early game, staying small and hiding while investing in cities, or expanding and slowly suffocating their opponents by gaining slightly more resources than they have. Using Generals also has several benefits compared to SC2. In Starcraft II game, a bot can easily have access to additional information that a human player cannot know and thus hard to evaluate the quality of a game agent. Another issue with SC2 is that in many games the bot can cheat by having extremely short reaction time which is impossible to achieve by a professional human players. Moreover, usually strategic games are very slow to simulate due to the large amount of information. However, Generals has a simple game board that is provided as a raw image (12x12) to both the agent and human players, as well as a small delay between turns that should lessen the effect of reaction time on the games. With our python version of generals that interfaces with OpenAI Gym BID1, we can achieve over 1000 frame per second with an average game time with a single GPU. This platform is considered more fexible and more transparent for evaluation. There are also resources available on the developer's website (http://dev.generals.io/), including an API to let an agent play online versus other bots or humans. They also have a collection of over 200,000 replays available as well as a Github repository full of parsing utilities as well as open-source bots online. In this work, we will have a high-level policy and a set of sub-policies to form a hierarchically structured agent. We will learn a series of sub-policies that can exploit one certain style in the game by self-play. The sub-policies must be strong because they are to be used by a High-level policy as a component. Moreover, the sub-policies must be diverse enough so that no one sub-policy dominates the rest. Then we train the high-level policy to choose from the sub-policies conditioned on observations. For both the sub-polices and the high-level policy in Generals, the observation includes a 12 by 12 map. In Section 3.1, we show the detailed architecture as well as PPO algorithm we use. In Section 3.2, we show how can we achieve the diverse and robust strategies that can exploit other strategies by using counter self play. And in Section 3.3, we show how to train the high-level policy. We adopt the Proximal Policy Optimization algorithm in our training framework. PPO uses a surrogate objective which is maximized while penalizing large changes to the policy. The algorithm alternates between sampling trajectories from the policy and performing SGD on the sampled dataset to optimize this surrogate objective. Although the algorithm is not new, we need to come up with specific network architecture to learn the policy and the value function. The policy network is a fully convolutional network. The rationale behind this selection of network architecture is that for real-time strategic games, there are multiple similar events will happen in different spatial location. The fully convolutional architecture has the capability of maintaining translation invariance and of generalizing similar events to different locations. We also share the initial 3 layers of parameters between the value network and the policy network. In this section, we describe how to achieve diverse and robust sub-policies by applying self-play. To learn a robust sub-policy by vanilla self-play requires occasional success by performing a series of random actions. But due to the large state space and action space, it is too challenging to win a game of Generals by chance on a large map. To alleviate this sparse signal problem in Generals, there are several methods such as engineering a dense reward for each step the agent take, imitating from demonstration replays as warm start or doing an exploration curriculum by adjusting the size of the map. We adopt the second method by imitating from demonstrations as a warm start. This method helps the sub-policy overcome the exploration problem, but it is still only rarely able to win against decent agents. A second problem is that if we start training a self-play agent, it will converge quickly to a local optima and stop generating diverse sub-policies which make this unusable for our high-level policy. In order to get a larger amount of diverse strategies we run conventional self-play with added an additional "achievement reward" as bonus for the initialization. However, we only use this to seed our method; after we get the initial policy, we only give all the agents the true rewards provided by the game and make the agent only want to win and win fast. During Counter Self-Play, we only train against the most recent self to exploit it instead of training against a set of previous selves as in BID0. We also reinitialize the agent's policy each time we save our current sub-policy and start a new one. This way, each agent has a fixed number of training iterations and the main difference between the subpolicies will be "style" not "strength". The current agent will become the counter strategy to the previous agent. After we have enough sub-policies, we can stop training and start the learning of high-level policy. In previous work BID5, there is another way to generate diverse policies by maximizing an information theoretic objective. However, the sub-policies generated with our method have meaning in that they are best-responses to different opponents rather than just being diverse in the sense that they reach different states in state-space. In this section, we describe how can we train a high-level policy based on the sub-policies we have. We initialize a parameterized high-level policy with random parameters. Then we train this agent with self-play as well. At each step in the game, the high-level policy will take the observation from the map as input and decide which sub-policy to choose. Different from prior work , we fix the sub-policies during training the high-level policy. After choosing, the sub-policy is fed with the same observation and execute the action based on it for a single step. On simple games such as iterated Rock-Paper-Scissor, the high-level policy can be computed once by first finding the pairwise expected payoffs of the sub-policies, and then solving a matrix game with Linear Programming as is the standard method in game theory. We test HASP and baseline methods on two discrete two-player competitive games: iterated Rock-Paper-Scissor (RPS) and Generals.1 We want to answer the following questions with our experiments.1. What types of sub-policies does our algorithm find? 2. Are the sub-policies from Counter Self-Play meaningfully diverse to be used by the high-level policy? 3. Can HASP lead to a stronger and more robust agent against opponents that are not seen in the training time compared with conventional self-play? For one baseline, we use conventional self-play, which is very similar to the way of how we get our sub-policies. To have a fair comparison with our method, we also incorporate the self-play agent with the imitation warm start stage described in section 3.2.We also compare with the baseline that randomly chooses from the time-averaged past sub-policies from counter self-play (CSP) as a baseline to see how much improvement we get by training a high-level policy instead of just choosing a sub-policy randomly. Iterated RPS is formulated simply as Rock-Paper-Scissor repeated over multiple turns. We consider two turns for illustration purposes. An agent gets a reward +1 for winning more turns than the opponent, -1 for less, and 0 for equal turns. Though the game has an obvious equilibrium strategy -uniformly random at any turn, sub-optimal players may incline towards specific patterns. For example, some players take the counter action in turn 2 against the opponent's action in turn 1 (the "counter" strategy), under the assumption that the opponent does not change his action ("repeat"). Some may reason further and assume the 1 Videos, codes and more information will be available at https://sites.google.com/view/hasp/home. Table 2: Performance on iterated RPS. R = rock_ only, P = paper_only, S = scissors_only, c = "counter", ci = "counter i", min denotes the minimum performance along a row winrate (opponent using "counter", so chooses the counter to that ("counter2") instead. The reasoning continues until it loops back to "counter6" = "repeat". If our algorithm is successful and the initial policy is ("repeat"), it should autonomously discover all these sub-policies in the strategy space. Running phase 1 of HASP here in ten different policies. We provide a t-SNE projection of the learned policies onto the 2D plane below. Since the action space of rock-paper-scissors is only three dimensional, we use the Linear Programming (LP) method to find a Nash-equilibrium mixed strategy where the moves consist of the ten learned sub-policies. The ing policy samples a sub-policy with probabilities shown in Table 1.In addition, we trained a baseline agent using conventional self-play. We found that using PPO to learn stochastic policies is quite hard, since most best-responses are pure strategies and when our agent reached low entropy, it would stop exploring, even if that policy was no longer good. In general, policies learned under self-play ed were close to deterministic and reasonable at exploiting a large number of different policies. However, we found that they were highly exploitable. Table 2 shows the performance of the different methods against some scripted test policies. Notice that the policy learned under HASP is less exploitable and therefore closer to an equilibrium policy than the different self-play runs. Note that randomly choosing a sub-policy also achieves a low exploitability, although not as low as with HASP. For all of our Generals experiments, we initialize our agent with behavioral cloning on an open-sourced agent called FloBot. After behavioral cloning, our agent can get only an average of -1.7 reward against FloBot but is able to win rarely. While we include our final against FloBot, note that they are slightly inflated in the self-play baseline, since the initial FloBot imitator is always in the training pool and therefore we learn a style that is effective against it. We find that when learning our our sub-policies, we can learn styles that are different from that of FloBot's. In order to break symmetry between strategies, we added a small negative reward to each time step. This discourages our agents from playing a safe, defensive strategy, and encourages them to use the information they know about their opponents in order to end the game faster. The final reward that we use for training in Generals is: We initialize the first phase of HASP with an agent trained to prefer owning territory on the top half of the map by adding the achievement reward described in Section 3.2. Running phase 1 of HASP, we obtained 6 different sub-policies that are diversely distributed in the sub-policy space. From observation, we find that the sub-policies are different from each other in terms of playing styles. We also find that, as we expected, each of them is trying to counter the previous self by a large margin, although it is exploitable itself. We performed the same test of training an agent against all our sub-policies to see if our sub-policies are meaningfully diverse. TAB2 shows the performance of our agent trained vs. several sub-policies generated via CSP compared to the best response to those policies. Since the best-responses have the highest reward, we conclude that knowing ahead of time which opponent we are facing is valuable and therefore our strategies are meaningfully diverse. For Generals, we trained our ensemble model using the Algorithm 1. The final usage frequencies (averaged over several games) of use of the sub-policies when facing another style of playing agent "expand_down" is shown in Table 5. Note that our agent learns to play a mixture of all strategies. Table 5: The frequency at which our policy selects each sub-policy when playing against expand_left sub-policies 1 2 3 4 5 6 0.14 0.2 0.18 0.22 0.12 0.14 TAB3 shows the performance of our HASP agent as well as the baselines against several held-out agents that we trained using self-play with an additional "achievement" reward: expand-left, expand-right, and expand-more. The agent trained using HASP achieves a higher reward against all of the held-out agents that we tested on. In contrast with our Iterated Rock-Paper-Scissors , randomly selecting a sub-policy at each turn is competitive with our policy that can change the probability of sampling a sub-policy with the current state. This suggests that the power of our method in Generals comes from the diverse sub-policies themselves. In this paper, we investigate a novel learning approach Hierarchical Agent with Self-Play to learning strategies in competitive games and real-time strategic games by learning several opponent-dependent sub-policies. We evaluate its performance on a popular online game, where we show that our approach generalizes better than conventional self-play approaches to unseen opponents. We also show that our algorithm vastly outperforms conventional self-play when it comes to learning optimal mixed strategies in simpler matrix games. Though our method has achieved good , there are some areas which could be improved in future research. In the future, we hope to also achieve good performance on larger versions of Generals, where games last longer and therefore learning is harder. We would also like to investigate further the effects that our algorithm has on exploration with sparse reward.

Represent the short summary for finding the corresponding scientific article (long)

We develop Hierarchical Agent with Self-play (HASP), a learning approach for obtaining hierarchically structured policies that can achieve high performance than conventional self-play on competitive real-time strategic games.

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems


Question: We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published and on the Billion Word benchmark, we achieve 23.02 perplexity. Language modeling is a basic task in natural language processing, with many applications such as speech recognition BID1 and statistical machine translation BID28 BID33 BID2. Recently, much progress has been made by neural methods BID3 BID20 ) based on LSTMs BID13, gated convolutional networks BID7 and self-attentional networks BID0.There are different choices for the basic unit we wish to model, including full words BID3, characters for the input BID15, or also the output BID18 as well as sub-words BID4 BID19. Word-based models are particularly challenging since computing probabilities for all 800K words of the BILLION WORD benchmark is still a substantial part of the overall computation BID6.A popular approach to lower the computational burden is to structure the output vocabulary so that not all probabilities need to be computed. The hierarchical softmax does this by introducing latent variables or clusters to simplify normalization BID8 BID22 BID21. This has been further improved by the adaptive softmax which introduces a variable capacity scheme for output word embeddings, assigning more parameters to frequent words and fewer parameters to rare words BID10.In this paper, we introduce adaptive input embeddings which extend the adaptive softmax to input word representations. This factorization assigns more capacity to frequent words and reduces the capacity for less frequent words with the benefit of reducing overfitting to rare words. For a competitive setup on the BILLION WORD benchmark, adaptive input embeddings reduce the number of parameters in the input and output layers by 23% while achieving higher accuracy over fixed size embeddings. When the adaptive input representations are tied with an adaptive softmax in the output, then the number of parameters is reduced by a total of 61%.Our experiments compare models based on word inputs, character inputs, as well as sub-word units using a self-attention architecture BID34. We show that models with adaptive word representations can outperform very strong character-based models while training more than twice as fast. We also substantially improve adaptive softmax by introducing additional dropout regularization in the tail projection. On the WIKITEXT-103 benchmark we achieve a perplexity of 18.7, a DISPLAYFORM0 A m e 4 R X e n K n z 4 r w 7 H 8 v R k l P s n M I f O J 8 / t g + R 9 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q u R U N k T x I l J t y S 1 E 3 C q d 9 P h a 6 P U = " > A DISPLAYFORM1 A m e 4 R X e n K n z 4 r w 7 H 8 v R k l P s n M I f O J 8 / t g + R 9 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q u R U N k T x I l J t y S 1 E 3 C q d 9 P h a 6 P U = " > A DISPLAYFORM2 A m e 4 R X e n K n z 4 r w 7 H 8 v R k l P s n M I f O J 8 / t g + R 9 g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q u R U N k T x I l J t y S 1 E 3 C q d 9 P h a 6 P U = " > A DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 DISPLAYFORM9 DISPLAYFORM10 Adaptive word representations are inspired by the adaptive softmax work BID10 which first described a GPU friendly way to construct a hierarchical softmax and showed that it performs very competitively compared to a full softmax, while offering significantly faster speed and a lower memory footprint. BID18 use a modified version of adaptive softmax which does not reduce the dimensionality of less frequent words in order to be able to share output embeddings with the input. This setup is akin to a hierarchical softmax with tied weights. We show that variable-sized input embeddings can perform better than fixed sized embeddings. Furthermore, this also enables weight sharing with an adaptive softmax output layer. BID18 evaluates both character-based and word-based factorizations but does not directly compare them to each other. We perform a direct comparison of word-based and characterbased input vocabularies and also compare to a sub-word factorization for both the input and output. Recently, BID0 demonstrated that self-attentional models can perform very well on language modeling tasks where the input and output is both characters. We also consider word-based benchmarks. The adaptive softmax exploits the fact that the distribution of word types in natural language follows a Zipfian distribution in order to improve the computation of the output probabilities. We apply the same intuition for input word embeddings with the motivation to reduce the number of parameters which frees up capacity for other parts of the model. We define a number of clusters that partitions the frequency ordered vocabulary V = V 1 ∪ V 2,..., V n−1 ∪ V n such that V i ∩ V j = ∅ for ∀i, j, and i = j, where V 1 contains the most frequent words and V n the least frequent words. We will refer to V 1 as the head and to any subsequent clusters loosely as tail. We reduce the capacity for each cluster by a factor of k. That is, if words in V 1 have dimension d, then words in V n have dimension d k n−1. We typically set k = 4 following BID10.Next, we add linear projections W 1 ∈ R d×d,..., W n ∈ R d/k n−1 ×d to map the embeddings of each cluster to dimension d so that the concatenated output of the adaptive input embedding layer can be easily used by the subsequent model FIG11. We also project V 1 which already has dimension d. When presented with a number of input words, the adaptive input embedding layer partitions the words into the various clusters, performs separate lookups in the embedding tables and then projects to dimension d, followed by concatenating the embeddings in the original order. Weight sharing. When the output layer is an adaptive softmax with the same partition of V, d, and k as the adaptive input layer, then we can tie the weights BID12 BID26. This further reduces the number of parameters and can simultaneously improve performance (§5). We can share both the parameters for the actual words as well as the projections W 1,..., W n Sharing the word embeddings is straightforward except for the head where the adaptive softmax has n − 1 additional embeddings for the remaining clusters which are not shared with the input. We share all projections, except for the head projection which is not available in the adaptive softmax since the model output is directly multiplied with the output word embeddings for the head band. Performance decreased when we added a head projection to the adaptive softmax in the output, regardless of when it was shared or not. Sharing both the word embeddings as well as the projections performed very well on WIKITEXT-103 but on BILLION WORD we only share the word embeddings as we found that this performed better on the validation set. We follow most of the architectural choices described in BID34 but use only a decoder network. We add sinusoidal position embeddings to the input layer and stack N = 16 blocks for both Each block contains two sub-blocks: the first is a multihead self-attention module with H = 16 heads. The second sub-block is a feed-forward module (FFN) of the form ReLU (W 1 X + b 1)W 2 + b 2 where W 1 ∈ R e×e f f, W 1 ∈ R e f f ×e and e = 1024, e f f = 4096 unless otherwise stated. Different to BID34 we apply layer normalization before the self-attention and FFN blocks instead of after, as we find it leads to more effective training. Sub-blocks are surrounded by a residual connection BID11.We use a dropout rate of 0.1 and attention dropout of 0.1 for BILLION WORD models, and increase regularization for by using dropout 0.3, and 0.1 ReLU dropout as well as attention dropout 0.1. We use the same hyperparameters for all models trained on the same dataset in order to enable a like for like comparison. When the dimensionality of the input or output layer differs from e, then we add a simple linear projection with no bias. We experiment on the BILLION WORD benchmark and WIKITEXT-103. BILLION WORD contains 768M word tokens and has a vocabulary of about 800K word types, which corresponds to words with more than 3 occurrences in the training set BID5.The training data of WIKITEXT-103 comprises about 100M tokens and a vocabulary of around 260K, corresponding to types with more than 3 occurrences in the training data BID17. The dataset is composed of shuffled Wikipedia articles where the context carries across sentences. For BILLION WORD we batch individual sentences since the corpus does not contain document structure. For we partition the training data into blocks of 512 contiguous tokens ignoring document boundaries. Evaluation is the same except that we require blocks to contain complete sentences totaling up to 512 tokens. We limit the number of tokens per GPU to a maximum threshold B per GPU. That is, we add examples of similar length until we reach this threshold. When we train on multiple GPUs, each GPU processes B tokens using the same model parameters. This increases the effective batch size to the product of the number of GPUs and B. For BILLION WORD models we use B = 2048 and typically train on 32 GPUs, giving an effective batch size of 65K tokens. The smaller vocabulary of enables increasing B to 4096 and we train on 8 GPUs. We found that large batch training is beneficial for this dataset and we therefore accumulate gradient updates over two batches before committing a parameter update BID23. This gives an effective batch size of 65K tokens for Embedding sizes. For fixed size word input layers and softmax output layers we generally use embeddings of size 512 for WIKITEXT-103. When we use an adaptive softmax in the output and fixed size word embeddings for the input, then we use dimension 256 for the input embeddings for BILLION WORD and 64 for We tuned this choice on the validation set (Appendix A). BPE inputs and outputs have embeddings of size 1024.Character CNN. We model character inputs by convolving the representations of all characters in a word following BID14 which applies several filters, then max pooling, a number of highway layers and a projection. Character embeddings have size 128 and we apply seven filters of size 1x128, 2x256, 3x384, 4x512, 5x512, 6x512, 7x512, where 3x128 indicates a filter processing three characters that outputs 128 features. We use a single highway layer for WIKITEXT-103, and two for BILLION WORD. We do not add start of word and end of word markers as they did not improve validation accuracy. We train on the same pre-processed data as the other models, with unknown tokens in both the inputs and outputs. Adaptive input representations and adaptive softmax. We use an adaptive softmax output layer to train models with large word-based vocabularies. For adaptive word inputs and adaptive softmax, we use embeddings of size d = 1024 for the head and reduce the size of subsequent clusters by a factor of k = 4. For WIKITEXT-103, we have three bands of size 20K (d=1024), 40K (d=256) and 200K (d=64). For BILLION WORD the bands are 60K (d=1024), 100K (d=256), and 640K (d=64).Sub-word models. We learn a byte-pair encoding (BPE) of 32K codes on the training data of each benchmark BID29. After applying the code to the training data we obtain a vocabulary of 33,337 tokens for and 32,347 tokens for BILLION WORD. BPE input/output embeddings have size 1024. The final evaluation is in terms word-level perplexity to be comparable to other models. The probability of a word is the product of the sub-word units. Different to BID34 we use Nesterov's accelerated gradient method BID32 ) with a momentum of 0.99 and we renormalize gradients if their norm exceeds 0.1 BID25. The learning rate is linearly warmed up from 10 −7 to 1 for 16K steps and then annealed using a cosine learning rate schedule with C cycles BID16. Each cycle runs for twice the number of updates than the previous cycle and we lower the maximum and minimum learning rates by a rate M compared to the previous cycle. The initial minimum learning rate is 10 −5 and the maximum is 1.BILLION WORD models train for a total of 975K updates over C = 3 cycles, the first cycle takes 137K steps, and we set M = 0.6. The WIKITEXT-103 models train for 286K steps over C = 4 cycles, the first cycle takes 18K setps and we set M = 0.75. We run experiments on DGX-1 machines with 8 NVIDIA V100 GPUs and machines are interconnected by Infiniband. We also use the NCCL2 library and the torch.distributed package for inter-GPU communication. We train models with 16-bit floating point precision, following BID24. 10 LSTMs + SNM10-SKIP 23.7 -- For the main on BILLION WORD, we doubled the batch size by training on 64 GPUs instead of 32 GPUs. We also consider two larger setups, one where we added four more blocks (N = 20) and increased the FFN dimension to e f f = 6144 (large), and another where we add another four blocks (N = 24) with e f f = 8192 and e = 1536 (very large). All other settings follow §4.4 and all models were trained for the same number of steps. TAB1 compares our models to previous work on BILLION WORD. The adaptive input model outperforms the best previously reported at an order of magnitude fewer parameters. Our large model performs nearly as well as an ensemble of over ten models and achieves a new state of the art of 24.14 perplexity. Our very large model performs as well as an ensemble of over ten models and achieves 23.02 perplexity. The Char-CNN model performs 0.6 PPL worse than the standard adaptive input model even though it trained for over 40% longer. TAB2 shows our on where adaptive inputs achieve 18.7 perplexity. For this only, we partition the training data into blocks of 3072 contiguous tokens instead of 512 tokens as for other experiments. During evaluation we require blocks to contain complete sentences totaling up to 3072 tokens of which the first 2560 tokens serve as context to score the last 512 tokens; we take care to score all tokens in the test and validation sets. We motivate this choice in §5.3. word units, both in the input and output, with embeddings of size 1024 (BPE) and shared weights (BPE-T). Next, we consider replacing the fixed size output representations by an adaptive softmax (ASM) and characters as input (CNN). Finally, we use both adaptive input word representations as well as an adaptive softmax (ADP) and a tied version (ADP-T). All models use the same selfattention architecture described in §4.1. TAB4 shows when training all configurations for the same number of updates. Adaptive input representations with tied input and output layers (ADP-T) achieve the highest accuracy at the same speed as the BPE models which have a very small vocabulary (33K versus 260K). CNN is 1 perplexity worse than ADP-T and requires well over twice the training time. It is the slowest approach, even though it has a fast adaptive softmax in the output. Fixed word embeddings perform least well (SM). Sub-word units are fast to train and perform better than word models with fixed sized embeddings. ASM improves over SM and greatly speeds up training. For ASM, we found that reducing the dimension of the input word embeddings to 64 on WIKITEXT-103 in better accuracy (Appendix A). TAB5 shows that adaptive input representations perform equally well on BILLION WORD compared to other factorizations. ADP-T is 34% faster than ADP because there are fewer parameters to update. Similar to before, ADP-T trains more than twice as fast as CNN at higher accuracy, however, the accuracy gap is narrower than for Regularization is more important on WIKITEXT-103 while models for BILLION WORD benefit from additional capacity. Because of this we used input word embeddings of size 256 for ASM. We also trained CNN without replacing input words outside the vocabulary by an unknown symbol, however, this only improved validation perplexity by 0.16. (cf. Figure 2). Next, we turn to the question of how well models perform on rare words compared to frequent words. We compute the average loss for each word in the test set and group words by frequency. Figure 2 shows on Tying weights helps all models on rare words, likely because of regularization effects. Fixed size word embeddings with a word softmax (SM and SM-T) do not perform well on rare words. This is likely due to underfitting on common words and we use the largest possible embedding size we could fit on 16GB GPU cards given our batch size (more experimentation in Appendix A). BPE and BPE-T perform poorly on rare words because probabilities are a product of several sub-word units. ADP-T performs best across all frequency ranges. Figure 3 bins the loss by the frequency of the previous word and shows that CNN does well when it has rare words in the context, however, ADP-T does best across all bins. Figure 4 shows an equivalent analysis for BILLION WORD. The largest differences between models is on rare words. CNN performs best on very rare words but is outperformed by ADP in all other settings. Similar to analysis (Appendix 5.3) binning the loss by the frequency of the previous word shows that weight sharing also helps for BILLION WORD and that CNN does very well on rare words for BILLION WORD compared to other models. TAB7 shows the importance of context size for Training block size is the number of consecutive tokens that are considered at once during training. Inference context is the number of tokens that are provided at evaluation before any tokens are scored. Simply increasing the training block size from 512 to 3072 in an improvement of nearly 1.2 perplexity with no inference context window. Increasing the context size at inference time in an improvement of 0.6 perplexity for the largest training block size. We also found that adaptive softmax can benefit from additional regularization of rare words. Adaptive softmax first projects the model output to the dimension of a particular cluster and then computes a dot product with the respective word embeddings. We add dropout to the output of the first projection for all clusters, except for the head. This change enables the adaptive softmax to outperform a standard softmax over fixed size output word embeddings on (Table 6).However, we found that adding dropout in this way is not helpful for larger datasets such as BILLION WORD. Unfortunately, a standard softmax over 800K words is not tractable and we were unable to make a comparison. It may be possible to achieve better by tuning dropout for each band of the tail and we leave this for future work. Table 6: Perplexity on WIKITEXT-103 when regularizing rare words in adaptive softmax. Adaptive input embeddings vary the size of input word embeddings which can improve accuracy while drastically reducing the number of model parameters. When sharing parameters with an adaptive softmax, the number of parameters can be further reduced which improves training speed. We presented a comparison between different input and output layer factorizations including word inputs, character inputs and sub-word units in both the input and output. Our experiments show that models with adaptive input embeddings train faster compared to character input CNNs while achieving higher accuracy. We achieve new state of the art on and BILLION WORD. In future work, we will apply variable sized input embeddings to other tasks. This appendix shows various ablation. TAB10 shows that reducing the capacity of fixed size word input embddings is beneficial on WIKITEXT-103. The next set of in TAB10 shows for various settings of the SM and SM-T models. We also experimented with sharing the head projection but found this to perform less well than not sharing it. Finally, TAB11 shows various band sizes for adaptive input word embbedings. We also show the performance of BID18 who use an adaptive softmax with equally sized word representations and share the input and output embeddings (no dim reduction, tied). Figure 3).

Represent the TLDR for finding the corresponding scientific article

Variable capacity input word embeddings and SOTA on WikiText-103, Billion Word benchmarks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.