Question: Represent the article for finding a single-sentence TLDR

Question: Knowledge Graph Embedding (KGE) has attracted more attention in recent years. Most of KGE models learn from time-unaware triples. However, the inclusion of temporal information beside triples would further improve the performance of a KGE model. In this regard, we propose LiTSE, a temporal KGE model which incorporates time information into entity/relation representations by using linear time series decomposition. Moreover, considering the temporal uncertainty during the evolution of entity/relation representations over time, we map the representations of temporal KGs into the space of multi-dimensional Gaussian distributions. The mean of each entity/ relation embedding at a time step shows the current expected position, whereas its covariance (which is stationary over time) represents its temporal uncertainty. Experiments show that LiTSE not only achieves the state-of- the-art on link prediction in temporal KGs, but also has the ability to predict the occurrence time of facts with missing time annotations, as well as the existence of future events. To the best of our knowledge, no other model is capable to perform all these tasks. Knowledge Graphs (KGs) are being used for gathering and organizing scattered human knowledge into structured knowledge systems. YAGO , NELL BID3, DBpedia BID0 and Freebase BID1 are among existing KGs that have been successfully used in various applications including question answering, assistant systems, information retrieval, etc. In these KGs, knowledge can be represented as RDF triples (s, p,o) in which s (subject) and o (object) are entities (nodes), and p (predicate) is the relation (edge) between them. KG embedding attempts to learn the representations of entities and relations in high-dimensional latent feature spaces while preserving certain properties of the original graph. Recently, KGE has become a very active research topic due to the wide ranges of downstream applications. Different KGE models have been proposed so far to efficiently learn the representations of KGs and perform KG completion as well as inferencing BID2 BID22 BID23 BID20 BID5.Most of existing KGE models solely learn from time-unknown facts and ignore the useful temporal information in KGs. In fact, there are many time-aware facts (or events) in some temporal KGs. For instance, (Obama, wasBornIn, Hawaii) happened at August 4, 1961, and (Obama, presidentOf, USA) was true from 2009 to 2017. These temporal KGs, e.g. ICEWS BID9, YAGO3 BID11, store such temporal information either explicitly or implicitly. Traditional KGE models such as TransE learn only from time-unknown facts and consequently cannot distinguish relations with similar semantic meaning. For instance, they often confuse relations such as wasBornIn and diedIn when predicting (person,?,location).To tackle this problem, Temporal KGE models BID4 BID6 BID18 encode time information in their embeddings. Temporal KGE models outperform traditional KGE models on link prediction over temporal KGs. It justifies that incorporation of time information can further improve the performance of a KGE model. Some existing temporal KGE models encode time information in a latent space e.g. representing time as a vector BID4 BID10. These models cannot capture some prop-erties of time information such as the length of time interval as well as order of two time points. Moreover, some exiting temporal graph embedding models BID18 BID19 consider the changes of entity representations over time as a kind of temporal evolution process, while they ignore the uncertainty during the temporal evolution. We argue that the evolution of entity representations has randomness, because the features of an entity at a certain time are not completely determined by the past information. For example, (Steve Jobs, diedIn, California) happened on 2011-10-05. The semantic characteristics of this entity should have a sudden change at this time point. However, due to the incompleteness of knowledge in KGs, this change can not be predicted only according to its past evolutionary trend. Therefore, the representation of Steve Jobs is supposed to include some random components to handle this uncertainty, e.g. a Gaussian noise component. To address the above problems, we propose a new temporal KGE model based on linear time series decomposition (LiTSE) that captures the evolution process of KG representations. LiTSE fits the evolution process of an entity or relation as a linear function of time with a Gaussian random noise. Inspired by, our approach represents each entity and relation as a multi-dimensional Gaussian distribution at each time step to introduce a random component. The mean of an entity/relation representation at a certain time step indicates its current expected position, which is obtained from its initial representation, its evolutionary direction vector which represents the long-term trend of its evolution and the current time. The covariance which describes the temporal uncertainty during its evolution, is denoted as a constant diagonal matrix for computing efficiency. Our contributions are as follows.• Learning the representations for temporal KGs is a relatively unexplored problem because most of existing KGE models only learn from time-unknown facts. We propose LiTSE, a new KGE model to incorporate the time information into the KG representations.• Different from the previous temporal KGE models which use time encoding to incorporate time information, LiTSE fits the evolution process of KG representations as a linear function of time. This enables us to observe and predict the time information directly from entity/relation representations. In particular, we can predict the occurrence of a fact in a future time, according to the known evolution trends of KG representations learned from the past information.• We specially consider the temporal uncertainty during the evolution process of KG representation. Thus, we model each entity as a Gaussian distribution at each time step and use KL-divergence between two Gaussian distributions to compute the scores of facts for optimization.• Beside performing link prediction in temporal KGs, our models are proved to be capable of estimating the occurrence time of a fact with missing time annotation, and predicting future events. The rest of the paper is organized as follows: Section 2 reviews related works. Our model is introduced in the section 3. The proposed model is evaluated and compared with state-of-the-art models in the section 4. Finally, the paper is concluded in the last section. A large amount of research has been done in KGE. These approaches can generally be categorized into two groups, namely semantic matching models and transnational distance models. RESCAL BID13 and its extensions, e.g. DistMult BID23, ComplEx BID20, ConvE BID5, are the semantic matching models. These models measure plausibility of facts by matching latent semantics of entities and relations embodied in their vector space representations. A few examples of translational distance models include TransE BID2, TransH BID22, TransD. These models measure the plausibility of a fact as the distance between the two entities, usually after a translation carried out by the relation. Particularly, KG2E takes into account the uncertainties of KG representations and represents entities and relations as random vectors drawn from multivariate Gaussian distributions. KG2E scores a fact by measuring the distance between the distributions of the entities and the relation. The above methods achieve good on link prediction in KGs. Moreover, some re-cent researches illustrate that the performances of KGE models can be further improved by incorporating time information in temporal KGs. TAE BID17 imposes temporal order constraints on time-sensitive relation pairs, e.g. BornIn and wasDiedIn, where the prior relation is supposed to lie close to the subsequent relation after a temporal transition. TAE only uses temporal order information between relations, but not the exact time information in facts. TTransE BID10 propose scoring functions which incorporate time representations into a TransE-type score function in different ways. BID6 utilizes recurrent neural networks to learn time-aware representations of relations and uses standard scoring functions from the existing KGE model, e.g. TransE BID2 and DistMult BID23. HyTE BID4 encodes time in the entity-relation space by associating a corresponding hyperplane to each timestamp. The above three methods represent each time step as a latent feature vector or a hyperplane matrix and update the entity/relation representations at different time steps with the corresponding time representations. That means the all entity/relation representations have the same evolution trend. In contrast, Know-Evolve BID18 ) models the temporal evolution of each entity representation as an individual temporal point process, which is a non-linear function of time. They exploit recurrent neural network to capture the dynamic characteristics of entity representations. Know-Evolve is also proved to perform well on time prediction. In this paper, we fit the temporal evolution of entity/relation representations by deploying linear time series decomposition. This enables us to directly observe and predict time information from entity/relation representations. We can also predict the future events according to the evolution trends of entity/relation representations. Moreover, inspired by KG2E, we map the entity and relation representations in a space of multi-dimensional Gaussian distributions to model the uncertainty of temporal KGs. Different from KG2E, we focus on the temporal embedding and give a specific definition of the uncertainty in temporal KGs, i.e. the randomness during the temporal evolution of KG representations. In this section, we present a detailed description of our proposed method, LiTSE, which not only uses relational properties between entities in triples but also incorporate the associated temporal meta-data by using linear time series decomposition. A time series is a series of time-oriented data. Time series analysis is widely used in many fields, ranging from economics and finance to managing production operations, to the analysis of political and social policy sessions BID12. An important technique for time series analysis is time series decomposition. This technique decomposes a time series into four components, including a trend component, a cyclical component, a seasonal component and an irregular component (i.e. "noise").In our method, we regard the evolution of an entity/relation representation as a linear time series and assume that it only consists of two components, i.e. a linear trend component and a Gaussian noise component. The motivation of this assumption is based on the following three points.• The simplicity of the embedding model architecture. Considering a temporal KG consisting of thousands of entities and relations, we can avoid introducing too many parameters by only using a trend component and a noise component to fit the temporal evolution of each relation/entity representation.• The capability of time prediction. Based on our proposed assumption, our model is capable to estimate the occurring time of a triple with the missing time annotation.• The efficiency of model training. Commonly, a moving-average model (MA model) is used when modeling the irregular term of a time series BID12. But we have to deploy a global optimization algorithm while training a MA model. Instead, we take a Gaussian noise as the irregular component in time series decomposition. This method enables us to employ mini-batch training for efficiency purposes. To incorporate temporal information into traditional KGs, a new temporal dimension is added to fact triples, denoted as a quadruple (s, p, o, t). It represents the creation of relationship edge p between subject entity s, and object entity o at time step t. The score term x spot = f t (e s, r p, e o) can represent the conditional probability or the confidence value of this event x spot, where e s, e o ∈ R Le, r p ∈ R Lr are representations of s, o and p. In the case of a fact (s, p, o, [t s, t e]), we consider it to be a positive triple for each time step between t s and t e. t s and t e denote the start and end time during which the triple (s, p, o) is valid. At each time step, the time-specific representations of an entity e i or a relation r i should be updated as e i,t or r i,t. In order to avoid information redundancy, we only incorporate time information into entity representations or relation representations, but not both. The model where time information is incorporated into relation representations is denoted as LiTSER. Another model with evolving entity representations is called as LiT-SEE. Thus, the score of a quadruple (s, p, o, t) can be represented as x spot = f e (e s,t, r p, e o,t) or x spot = f r (e s, r p,t, e o). Due to the similarity between LiTSEE and LiTSER, we take LiTSEE as an example to describe our method in this section. In our proposed model LiTSEE, we first utilize a linear function to fit the evolution processes of entity representations as: DISPLAYFORM0 where the e i is the time-independent latent representation of the ith entity which is subjected to ||e i || 2 = 1, the coefficient α i denotes its evolutionary rate, and the vector w i represents the direction of its evolution which is restricted to ||w i || 2 = 1. For LiTSEE, we use the following translationbased scoring function to measure the plausibility of a fact (s, p, o, t) BID2. DISPLAYFORM1 where ||r p || 2 = 1. Furthermore, to model the temporal uncertainty of the latent representations of the subject and the object in this fact, we assume e s,t and e o,t have randomness and obey Gaussian probability distributions: P s,t ∼ N (e s,t, Σ s) and P o,t ∼ N (e o,t, Σ o). Similarly, the predicate is represented as P r ∼ N (r p, Σ r). The mean vectors e and r, and covariance matrix Σ indicate the corresponding embedding representations for the Gaussian distribution. This advanced model based on LiTSEE is denoted as LiTSEE G.Similarly to LiTSEE, we consider the transformation of LiTSEE G from the subject to the object to be akin to the predicate in a positive fact. We use the following formula to express this transformation: P s,t − P o,t, which corresponds to the probability distribution P e,t ∼ N (µ e,t, Σ e). Here, µ e,t = e s,t − e o,t and Σ e = Σ s + Σ o. As a , combined with the probability of relation P r ∼ N (r p, Σ r), we measure the similarity between P e,t and P r to score the fact. KL divergence is a straightforward method of measuring the similarity of two probability distributions. We optimize the following score function based on the KL divergence between the entity-transformed distribution and relation distribution . DISPLAYFORM2 where, tr(Σ) and Σ −1 indicate the trace and inverse of the covariance matrix, respectively. Considering the simplified diagonal covariance, we can compute the trace and inverse of the matrix simply and effectively for LiTSEE G. The gradient of log determinant is et al., 2008). We can compute the gradients of Equation 3 with respect to the time-independent latent feature vectors, evolutionary direction vectors and covariance matrix (here acting as a vector) as follows: DISPLAYFORM3 DISPLAYFORM4 where DISPLAYFORM5 spot = Σ −1 r (r p + e o − e s + t(α o wIn the same way, we can extend LiTSER to LiTSER G by adding a Gaussian noise component into the linear evolution function of each relation/entity representation. The architectures of LiTSER and LiTSER G are similar to LiTSEE and LiTSEE G . The processes of computing gradients of the score functions in LiTSEE G and LiTSER G are also alike. Therefore, it is unnecessary to go into details about LiTSER and LiTSER G here. As mentioned in section 3.1, our proposed models are translation-based. Thus, we train our models by minimizing the margin-based ranking loss. DISPLAYFORM0 where, [T] is the set of time steps in the temporal KG, D + t is the set of positive triples with time stamp t and D − t is the set of negative examples. In this paper, we not only generate negative samples by randomly corrupting subjects or objects of the positives such as (s, p, o, t) and (s, p, o, t), but also add extra negative samples (s, p, o, t) which are present in the KG but do not exist in the subgraph for a particular time BID4. We use this time-dependent negative sampling approach for time prediction. In the other hand, to compare our model with baseline models fairly, we use uniform negative sampling method BID2 for link prediction and future event prediction. To avoid overfitting, we add some regularizations while learning the Gaussian embedding. As described in Section 3.1, the norms of the original representations of entities and relations, as well as the norms of all evolutionary direction vectors, are restricted by 1. Besides, the following constraint is considered for covariance when we minimize the loss L: DISPLAYFORM1 where, E and R are the set of entities and relations respectively, c min and c max are two positive constants. During training process, we use Σ l ← max(c min, min(c max, Σ l)) to achieve this regularization for diagonal covariance matrices. These constraints for the mean and covariance are also considered during initialization. To show the capability of LiTSE, we compare it and its extensions with other state-of-the-art baselines on link prediction. Particularly, we also evaluate our method for two other tasks: time prediction and future event prediction, which baseline models are not capable to handle. To compare our model with baselines, we used the following three datasets, namely ICEWS14, ICEWS05-15 and YAGO11K, released by Dasgupta et al. FORMULA0 and BID6. ICEWS14 and ICEWS05-15 are subsets of Integrated Crisis Early Warning System (ICEWS) BID9. ICEWS is a repository that contains political events with specific time annotations, e.g. The statistics of the datasets are listed in Table 1. We compare our method and other baselines by performing link prediction on ICEWS14, ICEWS05-15 and YAGO11k D. Specially, we also evaluate the performance of our proposed models on time prediction and future event prediction with two event-based datasets, ICEWS14 and ICEWS05-15. In this paper, we report the experimental on three tasks: Link Prediction, Time Prediction and Future Event Prediction. We split anew the facts into training, validation and test in a proportion of 80%/10%/10%. All facts in the test set occur after facts in the training/validation set. We train the model with the training set and judge whether quadruples in the test set are positive or not. The decision process is similar to triple classification BID15: for a fact (s, p, o, t), if x spot is below a relation-specific threshold δ r, then positive; otherwise negative. The thresholds δ r are determined on the validation set. For link prediction task, we compare our method with several state-of-the-art KGE models and existing time-wise KGE models, including TransE BID2, DistMult BID23, KG2E, ComplEx BID20, ConvE BID5, TTransE BID10,TA-TransE and TA-DistMult (García-Durán et al., 2018) as well as HyTE BID4. All these baselines are not applicable to estimating time information by computation (HyTE did it by ranking, which is much more time-consuming). Therefore, we compare the among our proposed models for time prediction. Considering that the above time-wise KGE models are not capable to represent a future time step, we compare our models with the above static KGE models for future event prediction. We implemented our models and baseline models in PyTorch, except TA-TransE and TA-DistMult. Since some implementation details of these two models were unclear, we report their from the original paper BID6. We used Adagrad optimizer to train all the implemented models and selected the optimal hyperparameters by early validation stopping according to MRR on the validation set. We restricted the iterations to 5000. For all the models, the batch size b = 512 was kept on both the datasets. We tuned the embedding dimensionalities d in {50, 100}, the learning rates lr in {0.001, 0.01, 0.1} and the ratio of negatives over positive training samples η in {1, 3, 5, 10}. For translation-based models, the margins γ were varied in the range {1, 2, 3, 5, 10}. For semantic matching models, the regularizer weights λ were chosen from the set {0.001, 0.01, 0.1}. For ConvE, we select dropout parameters from the set {0, 0.2}. Similar to the setting in KG2E, we selected the pair of restriction values c min and c max for covariance among {(0.005, 0.5), (0.01, 1), (0.03, 3), (0.05, 5)} for Gaussian embedding models. The default configuration for our proposed models is as follows: lr = 0.1, η = 10, γ = 1. Below, we only list the non-default parameters. For LiTSEE, the optimal configuration is as follows: lr = 0.01, γ = 10 on YAGO11k D. For Table 2: Link prediction (filtered setting). Rows 1-7: basic models with no time information. Rows 8-11: models which encode information. * indicates in this row were taken from (García-Durán et al., 2018). Dashes: could not be obtained. The best among all models are written bold. The red numbers are the best obtained from our implemention. LiTSEE G, the optimal configuration is as follows: γ = 2, (c min, c max) = (0.01, 1) on ICEWS14; (c min, c max) = (0.01, 1) on ICEWS05-15; lr = 0.01, γ = 10, (c min, c max) = (0.005, 0.5)] on YAGO11k D. For LiTSER, the optimal configuration is as follows: γ = 2 on ICEWS14; lr = 0.01, γ = 10 on YAGO11k D. For LiTSER G, the optimal configuration is as follows: (c min, c max) = (0.005, 0.5) both on ICEWS14 and ICEWS05-15;lr = 0.01, γ = 10, (c min, c max) = (0.005, 0.5)] on YAGO11k D. The above configurations were used for all three tasks. The obtained for different tasks are based on the above mentioned experimental setup. Table 2 shows the for link prediction task. In ICEWS14 and ICEWS05-15, LiTSEE G outperformed all embedding models considering MRR, Hits@10 and Hits@1. TransE implemented by BID6 got the best MR in these two datasets. It is noteworthy that the ratio of negatives over positive samples η used in (García-Durán et al., 2018) was 500, much higher than our setting. BID20 investigated the influence of η on KGE models and discovered that increasing η could lead to better . Thus, the obtained from BID6 would become worse if the same η as ours was used. Except the obtained from BID6 BID5 and BID20 showed that the performances of ConvE and ComplEx were remarkably better than KG2E on static KGs, e.g. FB15k and WN18 BID2. These prove that modeling temporal uncertainty in temporal KGs by mapping KG representations into the space of multi-dimensional Gaussian distribution substantially improve the performances of KGE models on temporal KGs. As mentioned in Section 3.2, we corrupted time information in positive facts to generate negative samples (s, p, o, t) for time prediction. TAB5 shows the of our proposed models for time prediction on ICEWS14 and ICEWS05-15. Beside Mean Absolute Errors (MAEs), we also report the proportions of testing examples which prediction errors are under 10 days, denoted as Error@10. Although MAEs of our models were high due to a small part of bad predictions, 57.5% of prediction errors of LiTSEE G on ICEWS14 were under 10 days, which proves the ability of our method for time prediction. We introduce LiTSE, a temporal KGE model that incorporates time information into KG representations by using linear time series decomposition. LiTSE fits the temporal evolution of KG representations over time as linear time series, which enables itself to estimate time information of a triple with the missing time annotation and predict the occurrence of a future event. Considering the uncertainty during the temporal evolution of KG representations, LiTSE maps the representations of temporal KGs into the space of multi-dimensional Gaussian distributions. The covariance of an entity/relation representation represents its randomness component. Experimental demonstrate that our method significantly outperforms the state-of-the-art methods on link prediction and future event prediction. Besides, our method can effectively predict the occurrence time of a fact. Our work establishes a previously unexplored connection between relational processes and time series analysis with a potential to open a new direction of research on reasoning over time. In the future, we will explore to use other time series analysis techniques to model the temporal evolution of KG representations. Along with considering the temporal uncertainty, another benefit of using time series analysis is to enable the embedding model to encode temporal rules. For instance, given two quadruple (s, p, o, t p) and (s, q, o, t q), there exists a temporal constraint t p < t q. Since the time information is represented as a numerical variable in a time series model, it is feasible to incorporate such temporal rules into our models. We will investigate the possibility of encoding temporal rules into our proposed models. DISPLAYFORM0 regularize the covariances for each entity and relation with constraint 6. 18. end loop TAB10 shows the statistics of datasets which are anew split for future event prediction, denoted as ICEWS14-F and ICEWS05-15F. As mentioned in Section 4.2, all of the facts in test set occur after the facts in training set and validation set, and the facts of validation set occur after the facts in training set. The time spans of training sets, validation sets and test sets of ICEWS14 and ICEWS05-15 are reported in TAB10. t e represents the end time of the dataset. For instance, t e of the training set of ICEWS14 is 2014/10/20 and t e of the validation set of ICEWS14 is 2014/11/22, which means the time stamps of quadruples in the validation set of ICEWS14 are between 2014/10/21 and 2014/11/22. In TAB1, we summarize the scoring function of baselines and our models and compare their space complexities. x, y, z = i x i y i z i denotes the tri-linear dot product; * denotes the convolution operator; Seq denotes a LSTM network; P t denotes the temporal projection for embeddings; w t denotes the embedding for the time step t. As shown in TAB1, our models have the same space complexities as traditional KGE models. On the other hand, the space complexities of TTransE and HyTE will be much higher than our models if n t is larger than n e and n r. Comparison of our models with baseline models for space complexity. n e, n r and n t are numbers of entities, relations and time steps. We borrow some notations from BID5 for simplicity.

Represent the TLDR for finding the corresponding scientific article

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the article for finding a single-sentence tldr

Question: We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines. Competitive games have been grand challenges for artificial intelligence research since at least the 1950s BID38 BID45 BID6. In recent years, a number of breakthroughs in AI have been made in these domains by combining deep reinforcement learning (RL) with self-play, achieving superhuman performance at ). In continuous control domains, competitive games possess a natural curriculum property, as observed in, where complex behaviors have the potential to emerge in simple environments as a of competition between agents, rather than due to increasing difficulty of manually designed tasks. Challenging collaborative-competitive multi-agent environments have only recently been addressed using end-to-end RL by BID21, which learns visually complex first-person 2v2 video games to human level. One longstanding challenge in AI has been robot soccer BID23, including simulated leagues, which has been tackled with machine learning techniques BID37 BID29 but not yet mastered by end-to-end reinforcement learning. We investigate the emergence of co-operative behaviors through multi-agent competitive games. We design a simple research environment with simulated physics in which complexity arises primarily through competition between teams of learning agents. We introduce a challenging multi-agent soccer environment, using MuJoCo BID46 which embeds soccer in a wider universe of possible environments with consistent simulated physics, already used extensively in the machine learning research community BID16 BID5 BID44. We focus here on multi-agent interaction by using relatively simple bodies with a 3-dimensional action space (though the environment is scalable to more agents and more complex bodies). 1 We use this environment to examine continuous multiagent reinforcement learning and some of its challenges including coordination, use of shaping rewards, exploitability and evaluation. We study a framework for continuous multi-agent RL based on decentralized population-based training (PBT) of independent RL learners BID20, where individual agents learn off-policy with recurrent memory and decomposed shaping reward channels. In contrast to some recent work where some degree of centralized learning was essential for multi-agent coordinated behaviors (e.g. BID28 BID9, we demonstrate that end-to-end PBT can lead to emergent cooperative behaviors in our soccer domain. While designing shaping rewards that induce desired cooperative behavior is difficult, PBT provides a mechanism for automatically evolving simple shaping rewards over time, driven directly by competitive match . We further suggest to decompose reward into separate weighted channels, with individual discount factors and automatically optimize reward weights and corresponding discounts online. We demonstrate that PBT is able to evolve agents' shaping rewards from myopically optimizing dense individual shaping rewards through to focusing relatively more on long-horizon game rewards, i.e. individual agent's rewards automatically align more with the team objective over time. Their behavior correspondingly evolves from random, through simple ball chasing early in the learning process, to more co-operative and strategic behaviors showing awareness of other agents. These behaviors are demonstrated visually and we provide quantitative evidence for coordination using game statistics, analysis of value functions and a new method of analyzing agents' counterfactual policy divergence. Finally, evaluation in competitive multi-agent domains remains largely an open question. Traditionally, multi-agent research in competitive domains relies on handcrafted bots or established human baselines BID21, but these are often unavailable and difficult to design. In this paper, we highlight that diversity and exploitability of evaluators is an issue, by observing non-transitivities in the agents pairwise rankings using tournaments between trained teams. We apply an evaluation scheme based on Nash averaging BID2 and evaluate our agents based on performance against pre-trained agents in the support set of the Nash average. We treat our soccer domain as a multi-agent reinforcement learning problem (MARL) which models a collection of agents interacting with an environment and learning, from these interactions, to optimize individual cumulative reward. MARL can be cooperative, competitive or some mixture of the two (as is the case in soccer), depending upon the alignment of agents' rewards. MARL is typically modelled as a Markov game BID39 ), which comprises: a state space S, n agents with observation and action sets O 1,..., O n and A 1,..., A n; a (possibly stochastic) reward function R i: S × A i → R for each agent; observation functions φ i: S → O i; a transition function P which defines the conditional distribution over successor states given previous state-actions: P (S t+1 |S t, A 1 t, ..., A n t), which satisfies the Markov property P (S t+1 |S τ, A 1 τ, ..., A n τ, ∀τ ≤ t) = P (S t+1 |S t, A Algorithm 1 Population-based Training for Multi-Agent RL.1: procedure PBT-MARL 2:{Ai} i∈ [1,..,N] N independent agents forming a population. for agent Ai in {Ai} i∈[1,..,N] do 4:Initialize agent network parameters θi and agent rating ri to fixed initial rating Rinit. Sample initial hyper-parameter θ h i from the initial hyper-parameter distribution. 6: end for 7:while true do 8:Agents play TrainingMatches and update network parameters by Retrace-SVG0. for match (si, sj) ∈ TrainingMatches do 10:UpdateRating (agents, according to some fitness function, inherit network parameters and some hyperparameters from stronger agents, with additional mutation. Hyperparameters can continue to evolve during training, rather than committing to a single fixed value (we show that this is indeed the case in Section 5.1). PBT was extended to incorporate co-play BID21 as a method of optimizing agents for MARL: subsets of agents are selected from the population to play together in multi-agent games. In any such game each agent in the population effectively treats the other agents as part of their environment and learns a policy π θ to optimize their expected return, averaged over such games. In any game in which π θ controls player i in the game, if we denote by π \i:= {π j} j∈{1,2,...,n},j =i the policies of the other agents j = i, we can write the expected cumulative return over a game as DISPLAYFORM0 where the expectation is w.r.t. the environment dynamics and conditioned on the actions being drawn from policies π θ and π \i. Each agent in the population attempts to optimize averaged over the draw of all agents from the population P, leading to the PBT objective J(π θ): DISPLAYFORM1, where the outer expectation is w.r.t. the probability that the agent with policy π θ controls player i in the environment, and the inner expectation is the expectation over the draw of other agents, conditioned on π θ controlling player i in the game. PBT achieves some robustness to exploitability by training a population of learning agents against each other. Algorithm 1 describes PBT-MARL for a population of N agents {A i} i∈ [1,..,N], employed in this work. Throughout our experiments we use Stochastic Value Gradients (SVG0) BID15 as our reinforcement learning algorithm for continuous control. This is an actor-critic policy gradient algorithm, which in our setting is used to estimate gradients network for bootstrapping, as is also described in BID13;. The identity of other agents π \i in a game are not explicitly revealed but are potentially vital for accurate action-value estimation (value will differ when playing against weak rather than strong opponents). Thus, we use a recurrent critic to enable the Q-function to implicitly condition on other players observed behavior, better estimate the correct value for the current game, and generalize over the diversity of players in the population of PBT, and, to some extent, the diversity of behaviors in replay. We find in practice that a recurrent Q-function, learned from partial unrolls, performs very well. Details of our Q-critic updates, including how memory states are incorporated into replay, are given in Appendix A.2. Reinforcement learning agents learning in environments with sparse rewards often require additional reward signal to provide more feedback to the optimizer. Reward can be provided to encourage agents to explore novel states for instance (e.g. BID4, or some other form of intrinsic motivation. Reward shaping is particularly challenging in continuous control (e.g. BID35 where obtaining sparse rewards is often highly unlikely with random exploration, but shaping can perturb objectives (e.g. BID1 ing in degenerate behaviors. Reward shaping is yet more complicated in the cooperative multi-agent setting in which independent agents must optimize a joint objective. Team rewards can be difficult to co-optimize due to complex credit assignment, and can in degenerate behavior where one agent learns a reasonable policy before its teammate, discouraging exploration which could interfere with the first agent's behavior as observed by BID12 . On the other hand, it is challenging to design shaping rewards which induce desired co-operative behavior. We design n r shaping reward functions {r j : S × A → R} j=1,...,nr, weighted so that r(·):= nr j=1 α j r j (·) is the agent's internal reward and, as in BID21, we use populationbased training to optimize the relative weighting {α j} j=1,...,nr. Our shaping rewards are simple individual rewards to help with exploration, but which would induce degenerate behaviors if badly scaled. Since the fitness function used in PBT will typically be the true environment reward (in our case win/loss signal in soccer), the weighting of shaping rewards can in principle be automatically optimized online using the environment reward signal. One enhancement we introduce is to optimize separate discount factors {γ j} j=1,...,nr for each individual reward channel. The objective optimized is then (recalling Equation 1) DISPLAYFORM0 This separation of discount factors enables agents to learn to optimize the sparse environment reward far in the future with a high discount factor, but optimize dense shaping rewards myopically, which would also make value-learning easier. This would be impossible if discounts were confounded. The specific shaping rewards used for soccer are detailed in Section 5.1. We simulate 2v2 soccer using the MuJoCo physics engine BID46. The 4 players in the game are a single sphere (the body) with 2 fixed arms, and a box head, and have a 3-dimensional action space: accelerate the body forwards/backwards, torque can be applied around the vertical axis to rotate, and apply downwards force to "jump". Applying torque makes the player spin, gently for steering, or with more force in order to "kick" the football with its arms. At each timestep, proprioception (position, velocity, accelerometer information), task (egocentric ball position, velocity and angular velocity, goal and corner positions) and teammate and opponent (orientation, position and velocity) features are observed making a 93-dimensional input observation vector. Each soccer match lasts upto 45 seconds, and is terminated when the first team scores. We disable contacts between the players, but enable contacts between the players, the pitch and the ball. This makes it impossible for players to foul and avoids the need for a complicated contact rules, and led to more dynamic matches. There is a small border around the pitch which players can enter, but when the ball is kicked out-of-bounds it is reset by automatic "throw in" a small random distance towards the center of the pitch, and no penalty is incurred. The players choose a new action every 0.05 seconds. At the start of an episode the players and ball are positioned uniformly at random on the pitch. We train agents on a field whose dimensions are randomized in the range 20m × 15m to 28m × 21m, with fixed aspect ratio, and are tested on a field of fixed size 24m × 18m. We show an example frame of the game in FIG0. We use population-based training with 32 agents in the population, an agent is chosen for evolution if its expected win rate against another chosen agent drops below 0.47. The k-factor learning rate for Elo is 0.1 (this is low, due to the high stochasticity in the game ). Following evolution there is a grace period where the agent does not learn while its replay buffer refills with fresh data, and a further "burn-in" period before the agent can evolve again or before its weights can be copied into another agent, in order to limit the frequency of evolution and maintain diversity in the population. For each 2v2 training match 4 agents were selected uniformly at random from the population of 32 agents, so that agents are paired with diverse teammates and opponents. Unlike multi-agent domains where we possess hand-crafted bots or human baselines, evaluating agent performance in novel domains where we do not possess such knowledge remains an open question. A number of solutions have been proposed: for competitive board games, there exits evaluation metrics such as Elo BID8 where ratings of two players should translate to their relative win-rates; in professional team sports, head-to-head tournaments are typically used to measure team performance; in BID0, survival-of-the-fittest is directly translated to multiagent learning as a proxy to relative agent performance. Unfortunately, as shown in BID2, in a simple game of rock-paper-scissors, a rock-playing agent will attain high Elo score if we simply introduce more scissor-play agents into a tournament. Survival-of-the-fittest analysis as shown in BID0 would lead to a cycle, and agent ranking would depend on when measurements are taken. Nash-Averaging Evaluators: One desirable property for multi-agent evaluation is invariance to redundant agents: i.e. the presence of multiple agents with similar strategies should not bias the ranking. In this work, we apply Nash-averaging which possesses this property. Nash-Averaging consists of a meta-game played using a pair-wise win-rate matrix between N agents. A row player and a column player simultaneously pick distributions over agents for a mixed strategy, aiming for a non-exploitable strategy (see BID2 .In order to meaningfully evaluate our learned agents, we need to bootstrap our evaluation process. Concretely, we choose a set of fixed evaluation teams by Nash-averaging from a population of 10 teams previously produced by diverse training schemes, with 25B frames of learning experience each. We collected 1M tournament matches between the set of 10 agents. FIG1 shows the pairwise expected goal difference among the 3 agents in the support set. Nash Averaging assigned nonzero weights to 3 teams that exhibit diverse policies with non-transitive performance which would not have been apparent under alternative evaluation schemes: agent A wins or draws against agent B on 59.7% of the games; agent B wins or draws against agent C on 71.1% of the games and agent C wins or draws against agent A on 65.3% of the matches. We show recordings of example tournament matches between agent A, B and C to demonstrate qualitatively the diversity in their policies (video 3 on the website 2). Elo rating alone would yield a different picture: agent B is the best agent in the tournament with an Elo rating of 1084.27, followed by C at 1068.85; Agent A ranks 5th at 1016.48 and we would have incorrectly concluded that agent B ought to beat agent A with a win-rate of 62%. All variants of agents presented in the experimental section are evaluated against the set of 3 agents in terms of their pair-wise expected difference in score, weighted by support weights. We describe in this section a set of experimental . We first present the incremental effect of various algorithmic components. We further show that population-based training with co-play and reward shaping induces a progression from random to simple ball chasing and finally coordinated behaviors. A tournament between all trained agents is provided in Appendix D. We incrementally introduce algorithmic components and show the effect of each by evaluating them against the set of 3 evaluation agents. We compare agent performance using expected goal difference weighted according to the Nash averaging procedure. We annotate a number of algorithmic components as follows: ff: feedforward policy and action-value estimator; evo: population-based training with agents evolving within the population; rwd shp: providing dense shaping rewards on top of sparse environment scoring/conceding rewards; lstm: recurrent policy with recurrent action-value estimator; lstm q: feedforward policy with recurrent action-value estimator; channels: decomposed action-value estimation for each reward component; each with its own, individually evolving discount factor. Population-based Training with Evolution: We first introduce PBT with evolution. FIG2 (ff vs ff + evo) shows that Evolution kicks in at 2B steps, which quickly improves agent performance at the population level. We show in FIG3 that Population-based training coupled with evolution yields a natural progression of learning rates, entropy costs as well as the discount factor. Critic learning rate gradually decreases as training progresses, while discount factor increases over time, focusing increasingly on long-term return. Entropy costs slowly decreases which reflects a shift from exploration to exploitation over the course training. Reward Shaping: We introduced two simple dense shaping rewards in addition to the sparse scoring and conceding environment rewards: vel-to-ball: player's linear velocity projected onto its unit direction vector towards the ball, thresholded at zero; vel-ball-to-goal: ball's linear velocity projected onto its unit direction vector towards the center of opponent's goal. Furthermore the sparse goal reward and concede penalty are separately evolved, and so can receive separate weight that trades off between the importance of scoring versus conceding. Dense shaping rewards make learning significantly easier early in training. This is reflected by agents' performance against the dummy evaluator where agents with dense shaping rewards quickly start to win games from the start FIG2, ff + evo vs ff + evo + rwd shp). On the other hand, shaping rewards tend to induce sub-optimal policies BID34 BID35; We show in FIG4 however that this is mitigated by coupling training with hyper-parameter evolution which adaptively adjusts the importance of shaping rewards. Early on in the training, the population as a whole decreases the penalty of conceding a goal which evolves towards zero, assigning this reward relatively lower weight than scoring. This trend is subsequently reversed towards the end of training, where the agents evolved to pay more attention to conceding goals: i.e. agents first learn to optimize scoring and then incorporate defending. The dense shaping reward vel-to-ball however quickly decreases in relative importance which is mirrored in their changing behavior, see Section 5.2.Recurrence: The introduction of recurrence in the action-value function has a significant impact on agents' performance as shown in FIG2 (ff + evo + rwd shp vs lstm + evo + rwd shp reaching weighted expected goal difference of 0 at 22B vs 35B steps). A recurrent policy seems to underperform its feedforward counterpart in the presence of a recurrent action-value function. This could be due to out-of-sample evaluators which suggests that recurrent policy might overfit to the behaviors of agents from its own population while feedforward policy cannot. Decomposed Action-Value Function: While we observed empirically that the discount factor increases over time during the evolution process, we hypothesize that different reward components require different discount factor. We show in FIG5 that this is indeed the case, for sparse environment rewards and vel-ball-to-goal, the agents focus on increasingly long planning horizon. In contrast, agents quickly evolve to pay attention to short-term returns on vel-to-ball, once they learned the basic movements. Note that although this agent underperforms lstm + evo + rwd shp asymptot- ically, it achieved faster learning in comparison (reaching 0.2 at 15B vs 35B). This agent also attains the highest Elo in a tournament between all of our trained agents, see Appendix D. This indicates that the training population is less diverse than the Nash-averaging evaluation set, motivating future work on introducing diversity as part of training regime. Assessing cooperative behavior in soccer is difficult. We present several indicators ranging from behavior statistics, policy analysis to behavior probing and qualitative game play in order to demonstrate the level of cooperation between agents. We provide birds-eye view videos on the website 2 (video 1), where each agent's value-function is also plotted, along with a bar plot showing the value-functions for each weighted shaping reward component. Early in the matches the 2 dense shaping rewards (rightmost channels) dominate the value, until it becomes apparent that one team has an advantage at which point all agent's value functions become dominated by the sparse conceding/scoring reward (first and second channels) indicating that PBT has learned a balance between sparse environment and dense shaping rewards so that positions with a clear advantage to score will be preferred. There are recurring motifs in the videos: for example, evidence that agents have learned a "cross" pass from the sideline to a teammate in the centre (see Appendix F for example traces), and frequently appear to anticipate this and change direction to receive. Another camera angle is provided on the website 2 (video 2) showing representative, consecutive games played between two fixed teams. These particular agents generally kick the ball upfield, avoiding opponents and towards teammates. Statistics collected during matches are shown in FIG6. The vel-to-ball plot shows the agents average velocity towards the ball as training progresses: early in the learning process agents quickly maximize their velocity towards the ball (optimizing their shaping reward) but gradually fixate less on simple ball chasing as they learn more useful behaviors, such as kicking the ball upfield. The teammate-spread-out shows the evolution of the spread of teammates position on the pitch. This shows the percentage of timesteps where the teammates are spread at least 5m apart: both agents quickly learn to hog the ball, driving this lower, but over time learn more useful behaviors which in diverse player distributions. pass/interception shows that pass, where players from the same team consecutively kicked the ball and interception, where players from the opposing teams kicked the ball in sequence, both remain flat throughout training. To pass is the more difficult behavior as it requires two teammates to coordinate whereas interception only requires one of the two opponents to position correctly. pass/interception-10m logs pass/interception events over more than 10m, and here we see a dramatic increase in pass-10m while interception-10m remains flat, i.e. long range passes become increasingly common over the course of training, reaching equal frequency as long-range interception. In addition to analyzing behavior statistics, we could ask the following: "had a subset of the observation been different, how much would I have changed my policy?". This reveals the extent to which an agent's policy is dependent on this subset of the observation space. To quantify this, we analyze counterfactual policy divergence: at each step, we replace a subset of the observation with 10 valid alternatives, drawn from a fixed distribution, and we measure the KL divergence incurred in agents' policy distributions. This cannot be measured for a recurrent policy due to recurrent states and we investigate ff + evo + rwd shp instead FIG2, where the policy network is feedforward. We study the effect of five types of counterfactual information over the course of training.ball-position has a strong impact on agent's policy distribution, more so than player and opponent positions. Interestingly, ball-position initially reaches its peak quickly while divergence incurred by counterfactual player/opponent positions plateau until reaching 5B training steps. This phase coincides with agent's greedy optimization of shaping rewards, as reflected in FIG7. Counterfactual teammate/opponent position increasingly affect agents' policies from 5B steps, as they spread out more and run less directly towards the ball. Opponent-0/1-position incur less divergence than teammate position individually, suggesting that teammate position has relatively large impact than any single opponent, and increasingly so during 5B-20B steps. This suggests that comparatively players learn to leverage a coordinating teammate first, before paying attention to competing opponents. The gap between teammate-position and opponents-position eventually widens, as opponents become increasingly relevant to the game dynamics. The progression observed in counterfactual policy divergence provides evidence for emergent cooperative behaviors among the players. Qualitatively, we could ask the following question: would agents coordinate in scenarios where it's clearly advantageous to do so? To this end, we designed a probe task, to test our trained agents for coordination, where blue0 possesses the ball, while the two opponents are centered on the pitch in front. A teammate blue1 is introduced to either left or right side. In Figure 9 we show typical traces of agents' behaviors (additional probe task video shown at Video 4 on our website 2): at 5B steps, pass intercept 5B left 0 100 5B right 31 90 80B left 76 24 80B right 56 27Figure 9: L1: Comparison between two snapshots (5B vs 80B) of the same agent. L2: number of successful passes and interception occurred in the first 100 timesteps, aggregated over 100 episodes.when agents play more individualistically, we observe that blue0 always tries to dribble the ball by itself, regardless of the position of blue1. Later on in the training, blue0 actively seeks to pass and its behavior is driven by the configuration of its teammate, showing a high-level of coordination. In "8e10 left" in particular, we observe two consecutive pass (blue0 to blue1 and back), in the spirit of 2-on-1 passes that emerge frequently in human soccer games. The population-based training we use here was introduced by BID21 for the capturethe-flag domain, whereas our implementation is for continuous control in simulated physics which is less visually rich but arguably more open-ended, with potential for sophisticated behaviors generally and allows us to focus on complex multi-agent interactions, which may often be physically observable and interpretable (as is the case with passing in soccer). Other recent related approaches to multi-agent training include PSRO BID24 and NFSP BID18, which are motivated by game-theoretic methods (fictitious play and double oracle) for solving matrix games, aiming for some robustness by playing previous best response policies, rather than the (more data efficient and parallelizable) approach of playing against simultaneous learning agents in a population. The RoboCup competition is a grand challenge in AI and some top-performing teams have used elements of reinforcement learning BID37 BID29, but are not end-to-end RL. Our environment is intended as a research platform, and easily extendable along several lines of complexity: complex bodies; more agents; multi-task, transfer and continual learning. Coordination and cooperation has been studied recently in deepRL in, for example, BID28 BID11 BID12 BID42; BID32, but all of these require some degree of centralization. Agents in our framework perform fully independent asynchronous learning yet demonstrate evidence of complex coordinated behaviors. BID0 introduce a MuJoCo Sumo domain with similar motivation to ours, and observe emergent complexity from competition, in a 1v1 domain. We are explicitly interested in cooperation within teams as well as competition. Other attempts at optimizing rewards for multi-agent teams include BID27. We have introduced a new 2v2 soccer domain with simulated physics for continuous multi-agent reinforcement learning research, and used competition between agents in this simple domain to train teams of independent RL agents, demonstrating coordinated behavior, including repeated passing motifs. We demonstrated that a framework of distributed population-based-training with continuous control, combined with automatic optimization of shaping reward channels, can learn in this environment end-to-end. We introduced the idea of automatically optimizing separate discount factors for the shaping rewards, to facilitate the transition from myopically optimizing shaping rewards towards alignment with the sparse long-horizon team rewards and corresponding cooperative behavior. We have introduced novel method of counterfactual policy divergence to analyze agent behavior. Our evaluation has highlighted non-transitivities in pairwise match and the practical need for robustness, which is a topic for future work. Our environment can serve as a platform for multiagent research with continuous physical worlds, and can be easily scaled to more agents and more complex bodies, which we leave for future research. In our soccer environment the reward is invariant over player and we can drop the dependence on i. SVG requires the critic to learn a differentiable Q-function. The true state of the game s and the identity of other agents π \i, are not revealed during a game and so identities must be inferred from their behavior, for example. Further, as noted in BID10, off-policy replay is not always fully sound in multi-agent environments since the effective dynamics from any single agent's perspective changes as the other agent's policies change. Because of this, we generally model Q as a function of an agents history of observations -typically keeping a low dimensional summary in the internal state of an LSTM: Q π θ (·, ·; ψ): X × A → R, where X denotes the space of possible histories or internal memory state, parameterized by a neural network with weights ψ. This enables the Q-function to implicitly condition on other players observed behavior and generalize over the diversity of players in the population and diversity of behaviors in replay, Q is learned using trajectory data stored in an experience replay buffer B, by minimizing the k-step return TD-error with off-policy retrace correction BID33, using a separate target network for bootstrapping, as is also described in BID13;. Specifically we minimize: DISPLAYFORM0 where ξ:= ((s t, a t, r t)) i+k t=i is a k-step trajectory snippet, where i denotes the timestep of the first state in the snippet, sampled uniformly from the replay buffer B of prior experience, and Q retrace is the off-policy corrected retrace target: DISPLAYFORM1 where, for stability,Q(·, ·;ψ): X ×A → R andπ are target network and policies BID30 periodically synced with the online action-value critic and policy (in our experiments we sync after every 100 gradient steps), and c s:= min(1, π(as|xs) β(as|xs) ), where β denotes the behavior policy which generated the trajectory snippet ξ sampled from B, and i s=i+1 c s:= 1. In our soccer experiments k = 40. Though we use off-policy corrections, the replay buffer has a threshold, to ensure that data is relatively recent. When modelling Q using an LSTM the agent's internal memory state at the first timestep of the snippet is stored in replay, along with the trajectory data. When replaying the experience the LSTM is primed with this stored internal state but then updates its own state during replay of the snippet. LSTMs are optimized using backpropagation through time with unrolls truncated to length 40 in our experiments. We use Elo rating BID8 ), introduced to evaluate the strength of human chess players, to measure an agent's performance within the population of learning agents and determine eligibility for evolution. Elo is updated from pairwise match and can be used to predict expected win rates against the other members of the population. For a given pair of agents i, j (or a pair of agent teams), s elo estimates the expected win rate of agent i playing against agent j. We show in Algorithm 3 the update rule for a two player competitive game for simplicity, for a team of multiple players, we use their average Elo score instead. By using Elo as the fitness function, driving the evolution of the population's hyperparamters, the agents' internal hyperparameters (see Section 3.3) can be automatically optimized for the objective we are ultimately interested in -the win rate against other agents. Individual shaping rewards would otherwise be difficult to handcraft without biasing this objective. We parametrize each agent's policy and critic using neural networks. Observation preprocessing is first applied to each raw teammate and opponent feature using a shared 2-layer network with 32 and 16 neurons and Elu activations BID7 to embed each individual player's data into a consistent, learned 16 dimensional embedding space. The maximum, minimum and mean of each dimension is then passed as input to the remainder of the network, where it is concatenated with the ball and pitch features. This preprocessing makes the network architecture invariant to the order of teammates and opponents features. Both critic and actor then apply 2 feed-forward, elu-activated, layers of size 512 and 256, followed by a final layer of 256 neurons which is either feed-forward or made recurrent using an LSTM BID19. Weights are not shared between critic and actor networks. We learn the parametrized gaussian policies using SVG0 as detailed in Appendix A, and the critic as described in Section A.2, with the Adam optimizer BID22 used to apply gradient updates. We also ran a round robin tournament with 50,000 matches between the best teams from 5 populations of agents (selected by Elo within their population), all trained for 5e10 agent steps -i.e. each learner had processed at least 5e10 frames from the replay buffer, though the number of raw environment steps would be much lower than that) and computed the Elo score. This shows the advantage of including shaping rewards, adding a recurrent critic and separate reward and discount channels, and the further (marginal) contribution of a recurrent actor. The full win rate matrix for this tournament is given in FIG0. Note that the agent with full recurrence and separate reward channels attains the highest Elo in this tournament, though performance against our Nash evaluators in Section 5.1 is more mixed. This highlights the possibility for non-transitivities in this domain and the practical need for robustness to opponents. To assess the relative importance of hyperparameters we replicated a single experiment (using a feed-forward policy and critic network) with 3 different seeds, see FIG0. Critic learning rate and entropy regularizer evolve consistently over the three training runs. In particular the critic learning rate tends to be reduced over time. If a certain hyperparameter was not important to agent performance we would expect less consistency in its evolution across seeds, as selection would be driven by other hyperparameters: thus indicating performance is more sensitive to critic learning rate than actor learning rate. Elo lstm + evo + rwd shp + channels 1071 lstm q + evo + rwd shp + channels 1069 lstm q + evo + rwd shp 1006 ff + evo + rwd shp 956 ff + evo 898Figure 10: Win rate matrix for the Tournament between teams: from top to bottom, ordered by Elo, ascending: ff + evo; ff + evo + rwd shp; lstm q + evo + rwd shp; lstm q + evo + rwd shp + channels; lstm + evo + rwd shp + channels. ELo derived from the tournament is given in the table. Figure 11: Hyperparameter evolution for three separate seeds, displayed over three separate rows. As well as the videos at the website 3, we provide visualizations of traces of the agent behavior, in the repeated "cross pass" motif, see FIG0.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce a new MuJoCo soccer environment for continuous multi-agent reinforcement learning research, and show that population-based training of independent reinforcement learners can learn cooperative behaviors

Introduce an approach to allow agents to learn  PPDDL action models incrementally over multiple planning problems under the framework of reinforcement learning.