Question: Represent the scientific article to find a short TLDR

Question: Current literature in machine learning holds that unaligned, self-interested agents do not learn to use an emergent communication channel. We introduce a new sender-receiver game to study emergent communication for this spectrum of partially-competitive scenarios and put special care into evaluation. We find that communication can indeed emerge in partially-competitive scenarios, and we discover three things that are tied to improving it. First, that selfish communication is proportional to cooperation, and it naturally occurs for situations that are more cooperative than competitive. Second, that stability and performance are improved by using LOLA , especially in more competitive scenarios. And third, that discrete protocols lend themselves better to learning cooperative communication than continuous ones. Multi-agent RL is used for many fully-competitive, zero-sum games (; ;) with the objective of finding a single, best learned agent (or team) at test time. In these games, a "best" agent is one that can outplay all opponents . We cannot evaluate its play by looking at the reward that agent achieves against a given opponent because the reward received will be a function of the choice of opponent. 1 In contrast, MARL in fully cooperative games has usually been able to make the assumption that we get to pick our team, and so can use self-play to try and achieve a maximal possible reward. (b). In this way, fully-cooperative MARL compares the maximum performance between learning algorithms and architectures by optimising the joint reward of the team. We investigate the space of partially competitive games known as general-sum games, where there is some amount of common interest and some amount of conflict. In this case, care must be taken in defining the "best" agent: it is not necessarily the agent that does as well or better than all opponents because, to achieve the highest possible reward in a general-sum game, agents might have to cooperate to some extent . For example, consider an agent playing iterated prisoner's dilemma. The agent that always defects will never have a reward worse than its opponent, but, when playing against a tit-for-tat agent, it will also not achieve the total reward of an agent that is also playing a tit-for-tat strategy. Since the maximum expected reward may only be possible by cooperating, agents must learn how to coordinate with each other. This can be done by training together or learning to understand opponent intentions at test time by observing their actions. The latter allows for ad-hoc comparison of learned agents at test time; the former requires comparing learning algorithms trained together. The latter should then require a sequential/iterative game, so there is time to infer opponent intentions before acting . It may also require meta-learning or other modifications to understand and adapt to these intentions as it seems current self-play methods are insufficient to adapt to ad-hoc play even against different versions of their own architecture . Work in general-sum MARL has mostly worked on analysing learning algorithms' ability to cooperate and resolve social dilemmas (a; ;) To date, investigations of emergent communication have remained mostly in the realm of fullycooperative games; ). For continuous communication, claimed to learn in mixed cooperative-competitive scenarios. However, their setup uses parameter sharing between opponents; their "mixed" case is non-competitive (and implicitly cooperative); their competitive game is actually two stages-one fully cooperative and one fully competitive; and, their in the competitive scenario are simply to mask out all communication. Previous attempts to learn discrete emergent protocols by selfish agents in competitive games have failed unless additional, more complex learning rules are adopted . In the latter case, they compare learning algorithms trained together as opposed to learned agents at test time to avoid a significant issue of comparing emergent communication agents-i.e., different protocols. Since an emergent protocol only has meaning between the agents that learned it together, comparing two learned agents at test time would require that they infer each others' protocols without training with them. This seems impossible, and it is more reasonable to frame it as a meta-learning problem where agents can have a brief adaptation period to synchronise their protocols. Meta-learning is beyond the scope of this paper, so we follow in comparing learning algorithms trained together. We also take a more principled approach than previous work by guaranteeing no communication through the action space, using more rigorous, quantitative criteria for evaluating communication, and precisely setting the levels of competitiveness. Notably, this work does not propose new architectures or learning rules but aims to take a critical look at existing beliefs and draw important distinctions as did for natural language emergence. Our experiments use the simple emergent-communication framework known as a sender-receiver game (or "referential game" ), which finds extensive use in economics and philosophy among others. In the classic game, the sender is given a target value to be communicated to the receiver via a message. The receiver receives the sender's message and must decode it to predict the target value. Both players are rewarded according to the negative of the receiver's prediction error. In this fully-cooperative setting, players often coordinate a protocol to transfer information as effectively as possible . The messages between the sender and the receiver can be categorised as "cheap talk": messages are costless, non-binding, non-verifiable and may affect the receiver's beliefs . Our work benefits greatly from -a seminal work in classical game theory. They study possible fixed communication equilibria under competition by giving the sender and receiver different targets and creating a conflict of interest. They perform a static analysis and prove the existence of a Nash equilibrium where the amount of information communicated is proportional to the alignment between the players' interests; however no informative equilibrium exists when interest diverge too greatly. In contrast, we do not look for existence of an equilibrium but do a dynamic analysis and show the feasibility of communication using standard learning rules in RL. We do not explicitly aim for equilibria but look at the information transfer of communication protocols in flux (and therefore out of equilibrium). This is more in line with previous work in emergent communication as well as evolutionary signalling . To investigate a range of competitive scenarios, we introduce a modified sender-receiver game with a continuous-bias variable, b, that represents the agents' conflict of interest, ranging from fully cooperative to fully competitive. The two players-the Sender (S) and the Receiver (R)-have corresponding targets (T s and T r), which are represented by angles on a circle that are b degrees apart: T r = (T s + b) mod 360 •. The game starts with the sender's target being sampled uniformly from the circle T s ∼ Uniform. The sender is given its target as input and outputs a message, m = S(T s), consisting of a single, discrete token from a vocabulary m ∈ V. The receiver is given the message and outputs a scalar action a = R(m). The goal of each agent is to make the receiver's action as close as possible to its own target value. After the receiver acts, both players get a loss between the action and their respective targets,. By using an L 1 loss between the angle of the target and action ) and a game with the maximum bias b = 180 • is fully competitive or constant-sum (a generalisation of zero-sum, see Appendix A.1 for proof). All values in-between, b ∈ (0 •, 180 •), represent the spectrum of partially cooperative/competitive general-sum games. Figure 1a gives an instance of this game; the game's algorithm is given in Algorithm 1 in Appendix B. This can be seen as the game from modified to cover the range of cooperative/competitive games. Both agents are implemented as MLPs with two hidden layers and ReLU nonlinearities between all layers. The targets are sampled from the circle, the sender takes its target, T s, as input and outputs a categorical distribution over a vocabulary from which we sample a message-its output. The receiver takes the message as input and deterministically outputs its action, a. Errors are calculated using the L 1 loss on the circle. The sender estimates its loss using the score function estimator -also known as REINFORCE -and has an added entropy regularisation term. Since the loss is differentiable with respect to the receiver, it is trained directly with gradient descent, so we are training in the style of a stochastic computation graph . We train for 30 epochs of 250 batches, with batch size 64, and set the circumference of our circle to 36 (so that a loss of 90 • is an error of 9). Both agents are trained using Adam . To evaluate, we use a fixed test set of 100 equidistant points and take the arg max of output distributions instead of sampling. We do all hyperparameter searches with Oríon , using random search with a fixed budget of searches. We perform a hyperparameter search to over both agents' learning rates, hidden layer sizes, the vocabulary size, and entropy regularisation (when used). We always report for given hyperparameters averaged over 5 random seeds, and we average our metric for hyperparameter search over the last 10 epochs to capture some level of stability as well as performance. All hyperparameter search spaces are available in the config files of the code repository. To evaluate the communication emerged with a cheap-talk channel, we can simply look to the sum of agents' L 1 losses. Under non-communication (or uninformative communication), we know that the receiver will just guess a point at random, and the average loss for both players is the expected value of the loss-given that it is drawn uniformly • Therefore, any error for either agent below 90 • is evidence of information transfer . Furthermore, since there is no other action space for agents to communicate in, the information transfer must be happening in the emergent communication space . Therefore, the lower L r 1 + L s 1 is, the more informative the learned protocol is; and, the most informative protocol will have the lowest loss min To show this comparison, we always plot the loss under uninformative communication (90 •) and the loss for each agent if they were to both fairly split the bias (b/2). While we have found evidence of information transfer, does that necessarily mean our agents have learned to communicate? For example, our hyperparameter search could find a minimal learning rate for the sender, such that it is essentially static, and a normal configuration for the receiver. The game would then become not one of learning a protocol between two agents, but rather just a receiver learning the sender's initial random mapping of targets to messages. The receiver could then dominate the sender by always choosing a = T r, which would yield L r 1 + L s 1 = b; namely, the optimal sum of losses and, therefore, optimal information transfer. This situation is clearly not what we are looking for, but it would be permissible, or potentially even encouraged, under an information-transfer objective (as measured by the sum of agents' L 1 losses). It is, therefore, necessary to delineate the differences in communication; here, we can look to extant in signalling . One perspective on information transfer is that of manipulation of receivers by senders or vice-versa ; this manifests as the domination of one agent over the other. We note that these situations are modelled as cue-reading or sensory manipulation, respectively, and are distinct from signalling-i.e., communication . Accordingly, communication requires both agents to receive a net benefit , which implies some degree of cooperation . For the fully-cooperative case, previous metrics of joint reward , or even influence of communication , are sufficient to drive the hyperparameter search. But for competitive scenarios, neither of these can distinguish between manipulation and cooperation . Since our focus is on the emergence of cooperative communication, we are looking for settings where both agents perform better than either their fully-exploited losses (L 2) as our hyperparameter-search metric. We can view our partially competitive scenario as having a common-interest loss (180 • − b), in which both agents are fully cooperative, and a conflict-of-interest loss (b), in which both agents are fully competitive. The sum of L 1 losses optimises only for the common interest, whereas L 2 prefers a more fair division of the conflict-of-interest loss in addition to optimising common interest (see proof in Appendix A.2). We use the L 2 metric only on hyperparameter search and keep L 1 as our game's loss to maintain a constant-sum game for the fully competitive case. • because the game is constant-sum and therefore trivially L •, but for completeness you can see the of a hyperparameter search with b = 180 • in Appendix C Figure 16. We report our in Figure 2 and find that agents do learn to cooperatively communicate without any special learning rules contrary to current literature. We can see that the performance decreases proportionately to the bias, meaning the sender is less informative with messages, forcing the receiver to be less accurate in its own guesses. This matches the theoretical of; information transfer with communication is inversely proportional to the conflict of interest. Plots for each b are in shown in Appendix C Figure 6. For the curve, we still plot the L 1 losses to maintain consistency and to make clearer the comparison to the no-communication baseline and the optimal information transfer (common interest maximisation). We find that our are basically unchanged between the different hyperparameter metrics; a relatively fair and useful protocol is learned by the agents, but this deteriorates in more competitive scenarios. This is clear when comparing the stability and relative efficacy of protocols in b = 30 •, 60 •, shown in Figures 2b, 2c, and that of b = 90 • shown in Figure 2d. We can understand this through the lens of honest communication, which can be taken advantage of in highly competitive scenarios. If, for example, the sender communicates, with complete honesty, its own coordinates, then the receiver can take advantage of this and choose its location exactly so that L r = 0 and L s = b. Comparing this situation to non-communication (L r = L s = 90 •), it is clear that even fully-exploited communication is a strictly dominant strategy for b < 90 • (i.e, when the game is more cooperative than competitive). For more competitive cases, fully-exploitable communication is no longer dominant, and active communication now requires both agents to cautiously cooperate. To achieve this cooperation, we • ] we show the training curve of the best hyperparameters found in 2b,2c,2d. We plot the test loss over training epochs and showing the mean and standard deviation over 5 seeds, finding that for b < 90 • we find stable and relatively fair communication is naturally learned propose using LOLA (a)-a learning rule, resembling theory-of-mind, that allows us to backpropogate through n steps of the opponent's learning. LOLA was able to emerge cooperative behaviour in an iterated prisoner's dilemma, so it is a prime candidate for resolving our game in a similar situation. We experiment with LOLA in three configurations-LOLA on the sender, LOLA on the receiver, LOLA on both-and do a similar hyperparameter search, with the added search space of the LOLA learning rate. Per the improvements made by , we replace the receiver's score function estimate with the DiCE estimator, and we backpropogate through exact copies of opponents as opposed to using opponent modelling. We show our in Figure 3a with extended plots in Figures 8, 9, 10 in Appendix C. We find that LOLA on the sender is ineffective, but LOLA on the receiver and on both agents does indeed lead to better performance. This implies that emerging communication in competitive scenarios necessitates cooperation and that this cooperation can be found through explicit opponent modelling. Furthermore, comparing the curves of basic agents (Figure 2d) with those of LOLA agents (Figure 3c) shows that gains in performance are not from one agent dominating the other, but from both agents improving and increasing stability. We also look at the performance of n-step LOLA, which backpropogates through n > 1 steps of opponent learning. Figure 3b demonstrates that 2-step LOLA slightly outperforms 1-step LOLA, but 3-step LOLA does not provide any increase over 2-step. We see from Figure 3d that the increase comes mostly from stability of learning and slight improvement on the part of the sender. • where our original setup does very poorly. 3b shows that higher step LOLA improves slightly further but not past 2-step. Best feasible communication protocols found for b = 90 • using 1-step (3c) and 2-step LOLA (3d) on both agents demonstrates that the gains in performance over the basic setup shown in Figure 2d are not just from one agent doing better (though the sender is doing better) but both agents improving in performance and stability. Shaded area is standard deviation over 5 seeds Another axis to consider is whether discrete or continuous communication lends itself better to learning with selfish agents. To compare, we make the sender's message a real-valued scalar and appropriately change its output distribution to be a Gaussian, for which it learns the mean and variance (concretely described in Algorithms 2 in Appendix B). We, again, run hyperparameter searches, and we consider training our baseline training with a REINFORCE Sender and deterministic receiver as well as training both agents with 1-step LOLA. Our in Figure 4a suggest that the learned protocols for continuous communication are all highly informative and near optimal. However, in all cases, the receiver is learning to manipulate the sender, and there is little evidence of cooperative communication. Indeed, we found no cases of both agents having a net benefit (L r, L s < 90 •) in any of the hyperparameter runs for continuous REINFORCE-deterministic agents past b = 90 •, and we only two cases of net benefit for LOLA-1 agents. Comparing this to discrete communication with the same LOLA-1 agents in Figure 4f, we can clearly see that they have a preference for more cooperative behaviour. Thus, we find that discrete messages are an important component in emerging cooperative self-interested communication. First and foremost, we show evidence against the current notion that selfish agents do not learn to communicate, and we hope our findings encourage more research into communication under The comparison between discrete and continuous communication for both the REINFORCE-deterministic setup as well as 1-step LOLA agents is shown in Figure 4a. We see that though overall continuous communication can achieve highest information transfer, the gains in performance seem to mostly from manipulation of the sender by the receiver. Two examples are shown for REINFORCE agents in Figures 4b,4c. To find a trend, we plot all 100 hyperparameter runs for b ∈ between continuous and discrete communication using 1-step LOLA agents in Figures 4d,4e,4f,4g. We find that manipulation is the common in continuous communication though individual cooperative points can sometimes be found. In general, continuous communication does not lend itself to cooperative communication competition. We have shown three important properties of communication. First, a game being more cooperative than competitive is sufficient to naturally emerge communication. Second, we've clarified the distinction between information transfer, communication, and manipulation, providing motivation for a better quantitative metric to measure emergent communication in competitive environments. Next, we've found that LOLA improves effective selfish communication and, using our metric, we find it does so by improving both agents' performance and stability. Finally, we've shown that using a discrete communication channel encourages the learning of cooperative commu-nication in contrast to the continuous communication channel setting, where we find little evidence of cooperation. In fully-cooperative emergent communication, both agents fully trust each other, so cooperatively learning a protocol is mutually beneficial. In competitive MARL, the task is using an existing protocol (or action space) to compete with each other. However, selfish emergent communication combines these two since the inherent competitiveness of using the protocol to win is tempered by the inherent cooperativeness of learning it; without somewhat agreeing to meanings, agents cannot use those meanings to compete . Thus, the agents must both learn a protocol and use that protocol simultaneously. In this way, even while competing, selfish agents emerging a communication protocol must learn to cooperate. A.1 PROOF OF FULLY COOPERATIVE/FULLY COMPETITIVE GAME For b = 0, T s = T r so trivially L s = L r and the game is fully cooperative. For b = 180 •, T r = T s + 180 mod 360 • we provide a visual demonstration in Figure 5 that the sum is always L s + L r = 180 • and therefore the game is constant-sum and fully competitive. We can also think of this as moving the actions distance d towards one agent's target means moving it distance d away from the other agent's target. •. Assume without loss of generality T s < T r so T r = T s + 180 Figure 5: The game with maximal bias 180 • showing the sum of L 1 losses L r + L s = 180 • We can extend the proof by symmetry (on the circle) for |T s − a| ≥ 360 • − |T s − a|, so the sum of losses L r + L s always equals 180 • so the game is constant-sum and therefore fully competitive. A.2 PROOF OF L 2 FAIRNESS Assume without loss of generality T s < T r, we are minimizing the sum of L 2 losses Sum of L 2 losses is minimized when the action is T s + b/2 or halfway between both agents' targets. Algorithm 1 Circular Biased Sender-Receiver Game

We manage to emerge communication with selfish agents, contrary to the current view in ML

represent the tldr for finding the corresponding scientific article
------
Example:
Provided: "In this paper, we proposed a model-free, off-policy IL algorithm for continuous control. Experimental results showed that our algorithm achieves competitive results with GAIL while significantly reducing the environment interactions." Match: "The goal of imitation learning (IL) is to enable a learner to imitate expert behavior given expert demonstrations. Recently, generative adversarial imitation learning (GAIL) has shown significant progress on IL for complex continuous tasks. However, GAIL and its extensions require a large number of environment interactions during training. In real-world environments, the more an IL method requires the learner to interact with the environment for better imitation, the more training time it requires, and the more damage it causes to the environments and the learner itself. We believe that IL algorithms could be more applicable to real-world problems if the number of interactions could be reduced. In this paper, we propose a model-free IL algorithm for continuous control. Our algorithm is made up mainly three changes to the existing adversarial imitation learning (AIL) methods – (a) adopting off-policy actor-critic (Off-PAC) algorithm to optimize the learner policy, (b) estimating the state-action value using off-policy samples without learning reward functions, and (c) representing the stochastic policy function so that its outputs are bounded. Experimental show that our algorithm achieves competitive with GAIL while significantly reducing the environment interactions. Recent advances in reinforcement learning (RL) have achieved super-human performance on several domains BID20 BID21. On most of such domains with the success of RL, the design of reward, that explains what agent's behavior is favorable, is obvious for humans. Conversely, on domains where it is unclear how to design the reward, agents trained by RL algorithms often obtain poor policies and behave worse than what we expect them to do. Imitation learning (IL) comes in such cases. The goal of IL is to enable the learner to imitate expert behavior given the expert demonstrations without the reward signal. We are interested in IL because we desire an algorithm that can be applied to real-world problems for which it is often hard to design the reward. In addition, since it is generally hard to model a variety of real-world environments with an algorithm, and the state-action pairs in a vast majority of realworld applications such as robotics control can be naturally represented in continuous spaces, we focus on model-free IL for continuous control. A wide variety of IL methods have been proposed in the last few decades. The simplest IL method among those is behavioral cloning (BC) BID23 which learns an expert policy in a supervised fashion without environment interactions during training. BC can be the first IL option when enough demonstration is available. However, when only a limited number of demonstrations are available, BC often fails to imitate the expert behavior because of the problem which is referred to compounding error BID25 -inaccuracies compound over time and can lead the learner to encounter unseen states in the expert demonstrations. Since it is often hard to obtain a large number of demonstrations in real-world environments, BC is often not the best choice for real-world IL scenarios. Another widely used approach, which overcomes the compounding error problem, is Inverse Reinforcement Learning (IRL) BID27 BID22 BID0 BID33. Recently, BID15 have proposed generative adversarial imitation learning (GAIL) which is based on prior IRL works. Since GAIL has achieved state-of-the-art performance on a variety of continuous control tasks, the adversarial IL (AIL) framework has become a popular choice for IL BID1 BID11 BID16. It is known that the AIL methods are more sample efficient than BC in terms of the expert demonstration. However, as pointed out by BID15, the existing AIL methods have sample complexity in terms of the environment interaction. That is, even if enough demonstration is given by the expert before training the learner, the AIL methods require a large number of state-action pairs obtained through the interaction between the learner and the environment 1. The sample complexity keeps existing AIL from being employed to real-world applications for two reasons. First, the more an AIL method requires the interactions, the more training time it requires. Second, even if the expert safely demonstrated, the learner may have policies that damage the environments and the learner itself during training. Hence, the more it performs the interactions, the more it raises the possibility of getting damaged. For the real-world applications, we desire algorithms that can reduce the number of interactions while keeping the imitation capability satisfied as well as the existing AIL methods do. The following three properties of the existing AIL methods which may cause the sample complexity in terms of the environment interactions:(a) Adopting on-policy RL methods which fundamentally have sample complexity in terms of the environment interactions.(b) Alternating three optimization processes -learning reward functions, value estimation with learned reward functions, and RL to update the learner policy using the estimated value. In general, as the number of parameterized functions which are related to each other increases, the training progress may be unstable or slower, and thus more interactions may be performed during training.(c) Adopting Gaussian policy as the learner's stochastic policy, which has infinite support on a continuous action space. In common IL settings, we observe action space of the expert policy from the demonstration where the expert action can take on values within a bounded (finite) interval. As BID3 suggests, the policy which can select actions outside the bound may slow down the training progress and make the problem harder to solve, and thus more interactions may be performed during training. In this paper, we propose an IL algorithm for continuous control to improve the sample complexity of the existing AIL methods. Our algorithm is made up mainly three changes to the existing AIL methods as follows:(a) Adopting off-policy actor-critic (Off-PAC) algorithm BID5 to optimize the learner policy instead of on-policy RL algorithms. Off-policy learning is commonly known as the promising approach to improve the complexity.(b) Estimating the state-action value using off-policy samples without learning reward functions instead of using on-policy samples with the learned reward functions. Omitting the reward learning reduces functions to be optimized. It is expected to make training progress stable and faster and thus reduce the number of interactions during training.(c) Representing the stochastic policy function of which outputs are bounded instead of adopting Gaussian policy. Bounding action values may make the problem easier to solve and make the training faster, and thus reduce the number of interactions during training. Experimental show that our algorithm enables the learner to imitate the expert behavior as well as GAIL does while significantly reducing the environment interactions. Ablation experimental show that (a) adopting the off-policy scheme requires about 100 times fewer environment interactions to imitate the expert behavior than the one on-policy IL algorithms require, (b) omitting the reward learning makes the training stable and faster, and (c) bounding action values makes the training faster. We consider a Markov Decision Process (MDP) which is defined as a tuple {S, A, T, R, d 0, γ}, where S is a set of states, A is a set of possible actions agents can take, T: S×A×S → is a transition probability, R: S×A → R is a reward function, d 0: S → is a distribution over initial states, and γ ∈ is a discount factor. The agent's behavior is defined by a stochastic policy π: S×A → and Π denotes a set of the stochastic policies. We denote S E ⊂ S and A E ⊂ A as sets of states and actions observed in the expert demonstration, and S π ⊂ S and A π ⊂ A as sets of those observed in rollouts following a policy π. We will use π E, π θ, β ∈ Π to refer to the expert policy, the learner policy parameterized by θ, and a behavior policy, respectively. Given a policy π, performance measure of π is defined as J (π, R) = E ∞ t=0 γ t R(s t, a t)|d 0, T, π where s t ∈ S is a state that the agent receives at discrete time-step t, and a t ∈ A is an action taken by the agent after receiving s t. The performance measure indicates expectation of the discounted return ∞ t=0 γ t R(s t, a t) when the agent follows the policy π in the MDP. Using discounted state visitation distribution denoted by ρ π (s) = ∞ t=0 γ t P(s t = s|d 0, T, π) where P is a probability that the agent receives the state s at time-step t, the performance measure can be rewritten as J (π, R) = E s∼ρπ,a∼π R(s, a). The state-action value function for the agent following π is defined as DISPLAYFORM0 |T, π, and Q π,ν denotes its approximator parameterized by ν. We briefly describe objectives of RL, IRL, and AIL below. We refer the readers to BID15 for details. The goal of RL is to find an optimal policy that maximizes the performance measure. Given the reward function R, the objective of RL with parameterized stochastic policies π θ: S×A → is defined as follows: DISPLAYFORM0 The goal of IRL is to find a reward function based on an assumption that the discounted returns earned by the expert behavior are greater than or equal to those earned by any non-experts behavior. Technically, the objective of IRL is to find reward functions R ω: S × A → R parameterized by ω that satisfies J (π E, R ω) ≥ J (π, R ω) where π denotes the non-expert policy. The existing AIL methods adopt max-margin IRL BID0 of which objective can be defined as follows: DISPLAYFORM1 2) The objective of AIL can be defined as a composition of the objectives and as follows: DISPLAYFORM2 The objective of Off-PAC to train the learner can be described as follows: DISPLAYFORM0 The learner policy is updated by taking the gradient of the state-action value. BID5 proposed the gradient as follows: provided another formula of the gradient using "re-parameterization trick" in the case that the learner policy selects the action as a = π θ (s, z) with random variables z ∼ P z generated by a distribution P z: DISPLAYFORM1 DISPLAYFORM2 3 ALGORITHMAs mentioned in Section.1, our algorithm (a) adopts Off-PAC algorithms to train the learner policy, (b) estimates state-action value without learning the reward functions, and (c) represents the stochastic policy function so that its outputs are bounded. In this section, we first introduce (b) in 3.1 and describe (c) in 3.2, then present how to incorporate (b) and (c) into (a) in 3.3. In this subsection, we introduce a new IRL objective to learn the reward function in 3.1.1 and a new objective to learn the value function approximator in 3.1.2. Then, we show that combining those objectives derives a novel objective to learn the value function approximator without reward learning in 3.1.3. We define the parameterized reward function as R ω (s, a) = log r ω (s, a), with a function r ω: S×A → parameterized by ω. r ω (s, a) represents a probability that the state-action pairs (s, a) belong to S E × A E. In other words, r ω (s, a) explains how likely the expert executes the action a at the state s. With this reward, we can also define a Bernoulli distribution p ω: Π×S×A → such that p ω (π E |s, a) = r ω (s, a) for the expert policy π E and p ω (π|s, a) = 1 − r ω (s, a) for any other policies π ∈ Π \ {π E} which include π θ and β. A nice property of this definition of the reward is that the discounted return for a trajectory {(s 0, a 0), (s 1, a 1),...} can be written as a log likelihood with p ω (π E |s t, a t): DISPLAYFORM0 Here, we assume Markov property in terms of p ω such that p ω (π E |s t, a t) for t ≥ 1 is independent of p ω (π E |s t−u, a t−u) for u ∈ {1, ..., t}. Under this assumption, the return naturally represents how likely a trajectory is the one the expert demonstrated. The discount factor γ plays a role to make sure the return is finite as in standard RL.The IRL objective can be said to aim at assigning r ω = 1 for state-action pairs (s, a) ∈ S E × A E and r ω = 0 for (s, a) ∈ S π × A π when the same definition of the reward R ω (s, a) = log r ω (s, a) is used. Following this fashion easily leads to a problem where the return earned by the non-expert policy becomes −∞, since log r ω (s, a) = −∞ if r ω (s, a) = 0 and thus log DISPLAYFORM1 The existing AIL methods seem to mitigate this problem by trust region optimization for parameterized value function approximator BID29, and it works somehow. However, we think this problem should be got rid of in a fundamental way. We propose a different approach to evaluate state-action pairs (s, a) ∈ S π × A π. Intuitively, the learner does not know how the expert behaves in the states s ∈ S \ S E -that is, it is uncertain which actions the expert executes in the states the expert has not visited. We thereby define a new IRL objective as follows: DISPLAYFORM2 where H denotes entropy of Bernoulli distribution such that: DISPLAYFORM3 Unlike the existing AIL methods, our IRL objective is to assign p ω (π E |s, a) = p ω (π|s, a) = 0.5 for (s, a) ∈ S π × A π. This uncertainty p ω (π E |s, a) = 0.5 explicitly makes the return earned by the non-expert policy finite. On the other hand, the objective is to assign r ω = 1 for (s, a) ∈ S E × A E as do the existing AIL methods. The optimal solution for the objective satisfies the assumption of IRL: DISPLAYFORM4, even though the objective does not aim at discriminating between (s, a) ∈ S E × A E and (s, a) ∈ S π × A π, As we see in Equation FORMULA7, the discounted return can be represented as a log likelihood. Therefore, a value function approximator Q π θ following the learner policy π θ can be formed as a log probability. We introduce a function q π θ,ν: S×A → parameterized by ν to represent the approximator Q π θ,ν as follows: DISPLAYFORM0 The optimal value function following a policy π satisfies the Bellman equation Q π (s t, a t) = R(s t, a t) + γE st+1∼T,at+1∼π Q π (s t+1, a t+1). Substituting π θ for π, log r ω (s t, a t) for R(s t, a t), and log q π θ,ν (s t, a t) for Q π (s t, a t), the Bellman equation for the learner policy π θ can be written as follows: DISPLAYFORM1 We introduce additional Bernoulli distributions P ν: Π × S × A:→ and P ωνγ: Π × S × A × S × A:→ as follows: DISPLAYFORM2 Using P ν and P ωνγ, the loss to satisfy Equation FORMULA13 can be rewritten as follows: DISPLAYFORM3 We use Jensen's inequality with the concave property of logarithm in Equation FORMULA4. Now we see that the loss L(ω, ν, θ) is bounded by the log likelihood ratio between the two Bernoulli distributions P ν and P ωνγ, and L(ω, ν, θ) = 0 if P ν (π E |s t, a t) = E st+1∼T,at+1∼π θ P ωνγ (π E |s t, a t, s t+1, a t+1).In the end, learning the approximator Q π θ,ν turns out to be matching the two Bernoulli distributions. A natural way to measure the difference between two probability distributions is divergence. We choose Jensen-Shannon (JS) divergence to measure the difference because we empirically found it works better, and thereby the objective to optimize Q π θ,ν can be written as follows: DISPLAYFORM4 where D JS denotes JS divergence between two Bernoulli distributions. Suppose the optimal reward function R ω * (s, a) = log r ω * (s, a) for the objective can be obtained, the Bellman equation FORMULA13 can be rewritten as follows: DISPLAYFORM0 Recall that IRL objective aims at assigning r ω * (s t, a t) = 1 for (s t, a t) ∈ S E × A E and r ω * (s t, a t) = 0.5 for (s t, a t) ∈ S π × A π where π ∈ Π \ {π E}. Therefore, the objective FORMULA9 is rewritten as the following objective using the Bellman equation FORMULA13: DISPLAYFORM1 Thus, r ω * can be obtained by the Bellman equation FORMULA6 as long as the solution for the objective can be obtained. We optimize q π θ,ν (s t, a t) in the same way of objective as follows: DISPLAYFORM2 We use P γ ν instead of P ωνγ in objective unlike the objective. Thus, we omit reward learning that the existing AIL methods require, while learning q π θ,ν (s t, a t) to obtain r ω *. Initialize time-step t = 0 and receive initial state s0 5:while not terminate condition do 6:Execute an action at = π θ (st, z) with z ∼ Pz and observe new state st+1 7:Store a state-action triplet (st, at, st+1) in B β. 8: DISPLAYFORM3 end while 10:for u = 1, t do 11:Sample mini-batches of triplets (s t ′ +1). end for 16: end for Recall that the aim of IL is to imitate the expert behavior. It can be summarized that IL attempts to obtain a generative model the expert has over A conditioned on states in S. We see that the aim itself is equivalent to that of conditional generative adversarial networks (cGANs) BID19. The generator of cGANs can generate stochastic outputs of which range are bounded. As mentioned in Section 1, bounding action values is expected to make the problem easier to solve and make the training faster. In the end, we adopt the form of the conditional generator to represent the stochastic learner policy π θ (s, z). The typical Gaussian policy and the proposed policy representations with neural networks are described in FIG0 Algorithm.1 shows the overview of our off-policy actor-critic imitation learning algorithm. To learn the value function approximator Q π θ,ν, we adopt a behavior policy β as π in the second term in objective We employ a mixture of the past learner policies as β and a replay buffer B β BID20 ) to perform sampling s t ∼ ρ π, a t ∼ π and s t+1 ∼ T. The buffer B β is a finite cache and stores the (s t, a t, s t+1) triplets in a first-in-first-out manner while the learner interacts with the environment. The approximator Q π θ,ν (s t, a t) = log q π θ,ν (s t, a t) takes (−∞, 0]. With the approximator, using the gradient to update the learner policy always punish (or ignore) the learner's actions. Instead, we adopt the gradient which directly uses Jacobian of Q π θ,ν.As do off-policy RL methods such as BID20 and, we use the target value function approximator, of which parameters are updated to track ν, to optimize Q π θ,ν. We update Q π θ,ν and π θ at the end of each episode rather than following each step of interaction. In recent years, the connection between generative adversarial networks (GAN) BID10 and IL has been pointed out BID15 BID6. BID15 show that IRL is a dual problem of RL which can be deemed as a problem to match the learner's occupancy measure BID31 to that of the expert, and that a choice of regularizer for the cost function yields an objective which is analogous to that of GAN. Their algorithm, namely GAIL, has become a popular choice for IL and some extensions of GAIL have been proposed BID1 BID11 BID16. However, those extensions have never addressed reducing the number of interactions during training. There has been a few attempts that try to improve the sample complexity in IL literatures, such as Guided Cost Learning (GCL) BID7. However, those methods have worse imitation capability in comparison with GAIL, as reported by BID8. As detailed in section 5, our algorithm have comparable imitation capability to GAIL while improving the sample complexity. Hester & Osband FORMULA7 proposed an off-policy algorithm using the expert demonstration. They address problems where both demonstration and hand-crafted rewards are given. Whereas, we address problems where only the expert demonstration is given. There is another line of IL works where the learner can ask the expert which actions should be taken during training, such as DAgger BID24, SEARN (Daumé &), SMILe BID25, and AggreVaTe BID26. As opposed to those methods, we do not suppose that the learner can query the expert during training. In our experiments, we aim to answer the following three questions: Q1. Can our algorithm enable the learner to imitate the expert behavior? Q2. Is our algorithm more sample efficient than BC in terms of the expert demonstration? Q3. Is our algorithm more efficient than GAIL in terms of the training time? To answer the questions above, we use five physics-based control tasks that are simulated with MuJoCo physics simulator BID32. See Appendix A for the description of each task. In the experiments, we compare the performance of our algorithm, BC, GAIL, and GAIL initialized by BC 23. The implementation details can be found in Appendix B. We train an agent on each task by TRPO BID28 using the rewards defined in the OpenAI Gym BID2, then we use the ing agent with a stochastic policy as the expert for the IL algorithms. We store (s t, a t, s t+1) triplets during the expert demonstration, then the triplets are used as training samples in the IL algorithms. In order to study the sample efficiency of the IL algorithms, we arrange two setups. The first is sparse sampling setup, where we randomly sample 100 (s t, a t, s t+1) triplets from each trajectory which contains 1000 triplets. Then we perform the IL algorithms using datasets that consist of several 100s triplets. Another setup is dense sampling setup, where we use full (s t, a t, s t+1) triplets in each trajectory, then train the learner using datasets that consist of several trajectories. If an IL algorithm succeeds to imitate the expert behavior in the dense sampling setup whereas it fails in the sparse sampling setup, we evaluate the algorithm as sample inefficient in terms of the expert demonstration. The performance of the experts and the learners are measured by cumulative reward they earned in a trajectory. We run three experiments on each task, and measure the performance during training. Figure 2 shows the experimental in both sparse and dense sampling setup. In comparison with GAIL, our algorithm marks worse performance on Walker2d-v1 and Humanoid-v1 with the datasets of the smallest size in sparse sampling setup, better performance on Ant-v1 in both setups, and competitive performance on the other tasks in both setups. Overall, we conclude that our algorithm is competitive with GAIL with regards to performance. That is, our algorithm enables the learner to imitate the expert behavior as well as GAIL does. BC imitates the expert behavior successfully on all tasks in the dense sampling setup. However, BC often fails to imitate the expert behavior in the sparse sampling setup with smaller datasets. Our algorithm achieves better performance than BC does all over the tasks. It shows that our algorithm is more sample efficient than BC in terms of the expert demonstration. Figure 3 shows the performance plot curves over validation rollouts during training in the sparse sampling setup. The curves on the top row in Figure 3 show that our algorithm denoted by Ours trains the learner more efficiently than GAIL does in terms of training time. In addition, the curves on the bottom row in Figure 3 show that our algorithm trains the learner much more efficiently than GAIL does in terms of the environment interaction. As opposed to BID15 suggestion, GAIL initialized by BC (BC+GAIL) does not improve the sample efficiency, but rather harms the leaner's performance significantly. We conducted additional ablation experiments to demonstrate that our proposed method described in Section.3 improves the sample efficiency. FIG2 shows the ablation experimental on Antv1 task. Ours+OnP, which denotes an on-policy variant of Ours, requires 100 times more interactions than Ours. The with Ours+OnP suggests that adopting off-policy learning scheme instead of on-policy one significantly improves the sample efficiency. Ours+IRL(D) and Ours+IRL(E) are variants of Ours that learn value function approximators using the learned reward function with the objective and FORMULA9, respectively. The with Ours+IRL(D) and Ours+IRL(E) suggests that omitting the reward learning described in 3.1 makes the training stable and faster. The with Ours+GP, which denotes a variant of Ours that adopts the Gaussian policy, suggests that bounding action values described in 3.2 makes the training faster and stable. The with Ours+DP, which denotes a variant of Ours that has a deterministic policy with fixed input noises, fails to imitate the expert behavior. It shows that the input noise variable z in our algorithm plays a role to obtain stochastic policies. In this paper, we proposed a model-free IL algorithm for continuous control. Experimental showed that our algorithm achieves competitive performance with GAIL while significantly reducing the environment interactions. A DETAILED DESCRIPTION OF EXPERIMENT TAB0 summarizes the description of each task, the performance of an agent with random policy, and the performance of the experts. We implement our algorithm using two neural networks with two hidden layers. Each network represents π θ and q ν. For convenience, we call those networks for π θ and q ω as policy network (PN) and Q-network (QN), respectively. PN has 100 hidden units in each hidden layer, and its final output is followed by hyperbolic tangent nonlinearity to bound its action range. QN has 500 hidden units in each hidden layer and a single output is followed by sigmoid nonlinearity to bound the output between. All hidden layers are followed by leaky rectified nonlinearity BID18. The parameters in all layers are initialized by Xavier initialization BID9. The input of PN is given by concatenated vector representations for the state s and noise z. The noise vector, of which dimensionality corresponds to that of the state vector, generated by zero-mean normal distribution so that z ∼ P z = N. The input of QN is given by concatenated vector representations for the state s and action a. We employ RMSProp BID14 for learning parameters with a decay rate 0.995 and epsilon 10 −8. The learning rates are initially set to 10 −3 for QN and 10 −4 for PN, respectively. The target QN with parameters ν ′ are updated so that ν ′ = 10 −3 * ν+(1−10 −3) * ν ′ at each update of ν. We linearly decrease the learning rates as the training proceeds. We set minibatch size of (s t, a t, s t+1) triplets 64, the replay buffer size |B β | = 15000, and the discount factor γ = 0.85. We sample 128 noise vectors for calculating empirical expectation E z∼Pz of the gradient. We use publicly available code (https://github.com/openai/imitation) for the implementation of GAIL and BC. Note that, the number of hidden units in PN is the same as that of networks for GAIL. All experiments are run on a PC with a 3.30 GHz Intel Core i7-5820k Processor, a GeForce GTX Titan GPU, and 32GB of RAM."

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.