{'fraction_non_alphanumeric': 0.02845253072679764, 'fraction_numerical': 0.006484080131617149, 'mean_word_length': 5.229053646775165, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Measuring collaborative emergent behavior in multi-agent reinforcement learning

Multi-agent reinforcement learning (RL) has important implications for the future of human-agent teaming. We show that improved performance with multi-agent RL is not a guarantee of the collaborative behavior thought to be important for solving multi-agent tasks. To address this, we present a novel approach for quantitatively assessing collaboration in continuous spatial tasks with multi-agent RL. Such a metric is useful for measuring collaboration between computational agents and may serve as a training signal for collaboration in future RL paradigms involving humans.


Introduction
Reinforcement learning (RL) is an attractive option for providing adaptive behavior in computational agents because of its theoretical generalizability to complex problem spaces [1,2]. In particular, deep RL recently produced striking results [3]. Extending RL to the multi-agent domain has received an increasing amount of attention as the need for human-agent teams has increased, especially with regards to training agents to behave collaboratively [1,[4][5][6][7][8].
Unfortunately, the nature of multi-agent RL makes guaranteeing collaboration between agents impossible except in limited provable cases, even when these methods yield better task performance [1,7]. Thus far, evaluating collaboration in multi-agent learning has been accomplished by measuring performance in tasks where coordination is required. While this may be satisfactory for discretized tasks where cooperative policies are provably optimal [1,[6][7][8] it is not clear that this methodology generalizes well to more complex and continuous tasks (such as those presented in [4,5] and here).
The question at hand is how to assess performance enhancing coordination (or collaboration) between agents in multi-agent RL tasks. Here, we present a method borrowed from the field of ecology, called convergence cross mapping (CCM), and show how it can be used to measure collaboration between agents during a predatorprey pursuit task. Additionally, we show a striking result: a state-of-the-art multi-agent reinforcement learning algorithm does not exhibit coordinated behavior between agents during a collaborative task even though high task performance is achieved, indicating that performance metrics alone are not sufficient for measuring collaboration.

Simulation Environment
In a modification of the classic predator-prey pursuit task (see Figure 1A-C), three slower predator agents score points each time they make contact with a prey agent in a continuous bounded 2D particle environment. Predator agents were identical in terms of capabilities (i.e., velocity and acceleration). This simulation environment was made available through the OpenAI Gym network [9] and was developed for the multi-agent deep learning algorithm discussed below [4]. Prey agents were capable of 33% greater acceleration and 25% greater maximum velocity than any predator agent, making capture by a solitary predator extremely difficult. All agents had the same mass, with minimal elastic properties to provide a small bump force upon collisions. Agent positions and velocities were randomized at the start of each episode, and acceleration was initially set to zero. Predator agents all received a fixed reward when any one of them made contact with the prey agent. Prey agents received a punishment when they were contacted.

Agents
In order to evaluate our metric's ability to estimate collaboration between predators in the predator-prey task, we utilized two types of agents: learning agents and fixedstrategy (non-learning) agents. Learning agents' behaviors were guided by a multiagent deep deterministic policy gradient (MADDPG) algorithm [4]. In all cases, prey agents were learning agents and thus utilized the MADDPG algorithm independent of predator behavior. Two types of distinct fixed-strategy predators were implemented to demonstrate upper and lower bounds of coordinated behaviors. The first (termed 'Chaser' predators, see Figure 1A) naively pursued prey agents by maximally accelerating in the instantaneous direction of the prey relative to the predator's own position. As such, Chasers were incapable of coordinating their behavior with each other.
The second strategy (termed 'Spring' predators, see Figure 1B) also naively minimized distance to the prey, but predator movements were modified by spring forces which constrained their position and velocity relative to one another. A Spring predator's movement direction was a sum of the spring forces acting on it and its desired vector of movement towards the prey. In this case, predator actions were explicitly coordinated with their partners'.
The MADDPG algorithm ( Figure 1C and 1E) used to guide learning agent behavior is an extension of a deep deterministic policy gradient (DDPG) algorithm [3] into the multi-agent domain [4]. Like DDPG, MADDPG utilizes an actor-critic model with deep neural networks representing policy and Q-learners ( Figure 1E). Multi-agent capabilities are achieved by passing information about each agent's state and actions to each critic network. As such, the learning agents are joint action learners as opposed to independent learners, giving them an advantage when coordinating their behaviors [8].
All conditions ( Figure 1A-C) were trained for 100k episodes to ensure model convergence before evaluating collaborative behavior ( Figure 1D). Each episode lasted for 25 time steps during learning and 2000 time steps during evaluation. These intervals were selected to match what was shown in literature for sufficient learning [4] and required for analysis [10]. During testing, 10 episodes were recorded without learning for each predator strategy in order to produce a distribution of agent behaviors with the same level of training over various random starting positions.

Collaboration Metrics
In order to measure collaboration, we used predators' positions over time as time-series data to implement a technique called convergent cross mapping (CCM) which examines the causal influence one time-series has on another [10]. The CCM technique embeds a time-series in a high-dimensional attractor space and then uses this embedded data as a model to predict states of the other time-series in its own attractor space. To the extent that this is possible, the original time-series is said to be causally driven by the time-series it attempts to model. Thus, in multi-agent tasks with homogeneous agents, we are defining collaboration to be the amount of causal influence between agents as measured by the CCM. Importantly, this metric can be misleading when two time-series have a mutually causal relationship with a third. For the present task, this is important because the causal influence one predator has on another can be confounded by their mutual relationship with the prey. To address this, at test time we modified the behavior of one predator agent such that its previous behavior was replaced by a secondary behavior that did not pursue the prey 1 . If a causal influence exists between predators, the movements of the modified predator should change the behavior of the other predators, even though the modified predator no longer pursues the prey (see Figure 1D). Table 1 shows the log scaled mean reward per episode for predator and prey agents in the Chaser, Spring, and MADDG cases. It is clear that MADDPG-equipped predators are betters performers of this pursuit task, achieving roughly an order of magnitude more average reward per trial. While the capacity for coordinated actions afforded by the MADDPG algorithm coupled with the increased task performance may suggest collaboration between predator agents, the CCM analysis tells a different story ( Figure 2). As shown, there is very little difference, in terms of causal influence, between MADDPG-equipped predators and naive Chasers. The causal influence of the modified agent might be marginally stronger for MADDPG predators (hinting at the shared action information afforded by this algorithm), but the effect is too weak to make any strong conclusions about the collaboration between the agents in this case.

Results
For Spring predators, on the other hand, there is a strong causal influence from the modified predator. The Spring experimental condition (see Figure 2) shows an example of coupled behaviors in the predator-prey pursuit task. This provides a strong validation for the CCM technique, and illustrates an ecological upper-bound on coordinated actions between learning agents. Fig. 2. CCM measure for causal influence of modified predator behavior on unmodified predators. True causal influence is indicated by an increase in CCM score (Y axis) as more of a trial (X axis) is used to build the model (so called "convergence" [10]

Discussion
The MADDPG algorithm presented by Lowe et al. [4] is promising, because it suggests that multi-agent deep RL may be able to solve complex problems in continuous tasks involving multiple actors. However, as we demonstrate here the improved performance of the MADDPG algorithm does not necessarily indicate coordinated behavior between agents.
Assessing collaboration between agents instead requires a direct measure of the coordination between agent actions. We present such a method here, in the form of CCM, and we validate the metric within the context of a continuous multi-agent task. Though we did not find strong evidence of coordination between the actions of learning predators, the marginal increase in coordination over Chaser predators points to potential for collaboration given sufficient learning pressure.
Producing collaborative behavior in computational agents is critical for the future of human-agent teams. Human-factors research has long held that when computational systems fail to adapt to human needs, serious issues in performance and human satisfaction can arise [11]. These issues are often best alleviated by promoting a collaborative relationship between humans and computational agents, rather than forcing humans to act as overseers [12,13]. Measures like CCM, which can be used to assess (and even promote) collaborative behaviors in multi-agent RL, constitute powerful tools for the future of human-computer interactions.