Question: Represent the article for finding a single-sentence TLDR

Question: Social dilemmas are situations where individuals face a temptation to increase their payoffs at a cost to total welfare. Building artificially intelligent agents that achieve good outcomes in these situations is important because many real world interactions include a tension between selfish interests and the welfare of others. We show how to modify modern reinforcement learning methods to construct agents that act in ways that are simple to understand, nice (begin by cooperating), provokable (try to avoid being exploited), and forgiving (try to return to mutual cooperation). We show both theoretically and experimentally that such agents can maintain cooperation in Markov social dilemmas. Our construction does not require training methods beyond a modification of self-play, thus if an environment is such that good strategies can be constructed in the zero-sum case (eg. Atari) then we can construct agents that solve social dilemmas in this environment. Bilateral cooperative relationships, where individuals face a choice to pay personal costs to give larger benefits to others, are ubiquitous in our daily lives. In such situations mutual cooperation can lead to higher payoffs for all involved but there always exists an incentive to free ride. In a seminal work BID3 asks a practical question: since social dilemmas are so ubiquitous, how should a person behave when confronted with one? In this work we will take up a variant of that question: how can we construct artificial agents that can solve complex bilateral social dilemmas? First, we must define what it means to'solve' a social dilemma. The simplest social dilemma is the two player, repeated Prisoner's Dilemma (PD). Here each player chooses to either cooperate or defect each turn. Mutual cooperation earns high rewards for both players. Defection improves one's payoff but only at a larger cost to one's partner. For the PD, BID2 suggest the strategy of tit-for-tat (TFT): begin by cooperating and in later turns copy whatever your partner did in the last turn. TFT and its variants (eg. Win-Stay-Lose-Shift, BID37) have been studied extensively across many domains including the social and behavioral sciences, biology, and computer science. TFT is popular for several reasons. First, it is able to avoid exploitation by defectors while reaping the benefits of cooperation with cooperators. Second, when TFT is paired with other conditionally cooperative strategies (eg. itself) it achieves cooperative payoffs. Third, it is error correcting because after an accidental defection is provides a way to return to cooperation. Fourth, it is simple to explain to a partner and creates good incentives: if one person commits to using TFT, their partner's best choice is to cooperate rather than try to cheat. Our contribution is to expand the idea behind to TFT to a different environment: one shot Markov social dilemmas that require function approximation (eg. deep reinforcement learning). We will work with the standard deep RL setup: at training time, our agent is given access to the Markov social dilemma and can use RL to compute a strategy. At test time the agent is matched with an unknown partner and gets to play the game with that partner once. We will say that the agent can solve a social dilemma if it can satisfy the four TFT properties listed above. We call our strategy approximate (because we use RL function approximation) Markov (because the game is Markov) tit-for-tat (amTFT) which we show can solve more complex Markov social dilemmas. The first issue amTFT needs to tackle is that unlike in the PD'cooperation' and'defection' are no longer simple labeled strategies, but rather sequences of choices. amTFT uses modified self-play 1 to learn two policies at training time: a fully cooperative policy and a'safe' policy (we refer to this as defection). The second issue is that we are considering a setup where our agent will only play the social dilemma once at test time. Thus the goal of amTFT is to intelligently switch between the learned policies within a single game.3 amTFT performs this as follows: at each time step during test time the amTFT agent computes the gain from the action their partner actually chose compared to the one prescribed by the cooperative policy. This can be done either using a learned Q function or via policy rollouts. We refer to this as a per period debit. If the total debit is below a threshold amTFT behaves according to the cooperative policy. If the debit is above the threshold, the agent switches to the defecting policy for k turns and then returns to cooperation. This k is computed such that the partner's gains (debit) are smaller than the losses they incur (k lost turns of cooperation).We show both analytically and experimentally that amTFT can solve Markov social dilemmas (in the Axelrod sense defined above). Our experiments using a grid-world, Coins, and a modification of an Atari game where players must learn from pixels, the Pong Player's Dilemma also demonstrate that an important component of amTFT is defining a partner's'defection' in terms of value and not actions. This choice makes amTFT robust to a partner using one of a class of outcome-equivalent cooperative policies as well function approximation, important properties for scaling agents beyond simple games. We note that for the purposes of this paper we define the'cooperative' the policies as the ones which maximize the sum of both players' payoff. This definition seems natural for the case of the symmetric games we study (and is the one that is typically used in eg. the literature on the evolution of cooperation). However, it is well known that human social preferences take into account distribution (eg. inequity BID15), various forms of altruism BID1 BID44, and context dependent concerns (eg. social norms, see BID46 ; BID23 ; BID43 for how social norms affect economic games and can be manipulated in the lab). Thus when applying amTFT in other circumstances the correct'focal point' needs to be chosen. The automatic determination of focal points is an important topic for future research but far beyond the scope of this paper. However, we note that once this focal point is determined the amTFT algorithm can be used exactly as in this paper simply by swapping out the cooperative objective function during training time. A large literature on the'folk theorem' asks whether in a repeated game there exists an equilibrium which maintains cooperative payoffs using strategies which take as input histories of observations BID19 BID13 and output stage-game actions. A computer science branch of this literature asks whether it is possible to compute such equilibria either in repeated matrix games BID32 or in repeated Markov games BID4. These works are related to our questions but have two key differences: first, they focus on switching strategies across iterations of a repeated game rather than within a single game. Second, perhaps more importantly, this literature focuses on finding equilibria unlike the Axelrod setup which focuses on finding a'good' strategy for a single agent. This difference in focus is starkly illustrated by TFT itself because both agents choosing TFT is not an equilibrium (since if one agent commits to TFT the partner's best response is not TFT, but rather always cooperate). 1 We note that one advantage of amTFT is that it requires no additional machinery beyond what is required by standard self-play, thus if we can construct competitive agents in some environment (eg. Atari, BID34) then we can also construct agents that solve social dilemmas in that environment.2 In the PD this action is'defect' but in most real social dilemmas this is not the case. For example social dilemmas occur naturally in economic situations where agents face a choice of trading and realizing gains from trade or simply producing everything they need on their own. In this case a safe policy is the outside option of'stop transacting with this agent.'3 The focus on a single test game is a way in which the problem we consider differs from what is normally studied in the literature on maintaining cooperation in repeated games BID19 BID32 BID4. In a standard'folk theorem' setup agents play a game repeatedly and maintain cooperation in one iteration of a game by threats of defection in the next iteration. A second related literature focuses on learning and evolution in games BID18 BID47 BID51 BID38 BID9 with recent examples applying deep learning to this question BID41. Though there is a large component of this literature focusing on social dilemmas, these works typically are interested how properties of the environment (eg. initial states, payoffs, information, learning rules used) affect the final state of a set of agents that are governed by learning or evolutionary dynamics. This literature gives us many useful insights, but is not usually focused on the question of design of a single agent as we are. A third literature focuses on situations where long term interactions with the same partner means that a good agent needs to either to discern a partner's type BID31 or be able shape the adaptation of a learning partner BID4 BID17. BID4 use reward shaping in the Prisoner's Dilemma to construct'leader' agents that convince'followers' to cooperate and BID17 uses a policy gradient learning rule which includes an explicit model of the partner's model. These works are related to ours but deal with situations where interactions are long enough for the partner to learn (rather than a single iteration) and require either explicit knowledge about the game structure BID4 or the partner's learning rule BID17.There is a recent surge of interest in using deep RL to construct agents that can get high payoffs in multi-agent environments. Much of this literature focuses either on zero-sum environments (; BID52 BID8 BID24 ;) or coordination games without an incentive to defect BID33 BID16 BID45; BID42 BID28 BID11 BID14 BID22 ) and uses self-play to construct agents that can achieve good outcomes. 4 We show that in the presence of social dilemmas applying this self-play approach naively often leads to bad outcomes. Finally, there is a large literature using the repeated PD to study human decision-making in social dilemmas BID20 BID7. In addition, recent work in cognitive science has begun to use more complex games and RL techniques quite related to ours . However, while this work provides useful insights into potentially useful strategies the main objective of this work is to understand human decision-making, not to actively improve the construction of agents. We now turn to formalizing our main idea. We will work with a generalization of Markov decision problems:Definition 1 BID49 ) A (finite, 2-player) Markov game consists of a set of states S = {s 1, . . ., s n}; a set of actions for each player DISPLAYFORM0 which tells us the probability distribution on the next state as a function of current state and actions; a reward function for each player R i: S × A 1 × A 2 → R which tells us the utility that player gains from a state, action tuple. We assume rewards are bounded. Players can choose between policies which are maps from states to probability distributions on actions π i: S → ∆(A i). We denote by Π i the set of all policies for a player. Through the course of the paper we will use the notation π to refer to some abstract policy andπ to learned approximations of it (eg. the output of a deep RL procedure).Definition 2 A value function for a player i inputs a state and a pair of policies V i (s, π 1, π 2) and gives the expected discounted reward to that player from starting in state s. We assume agents discount the future with rate δ which we subsume into the value function. A related object is the Q function for a player i inputs a state, action, and a pair of policies Q i (s, π 1, π 2) and gives the expected discounted reward to that player from starting in state s taking action a and then continuing according to π 1, π 2 afterwards. We will be talking about strategic agents so we often refer to the concept of a best response: Definition 3 A policy for agent j denoted π j is a best response starting at state s to a policy π i if for any π j and any s along the trajectory generated by these policies we have DISPLAYFORM1 We denote the set of such best responses as BR j (π i, s). If π j obeys the inequality above for any choice of state s we call it a perfect best response. The set of stable states in a game is the set of equilibria. We call a policy for player 1 and a policy for player 2 a Nash equilibrium if they are best responses to each other. We call them a Markov perfect equilibrium if they are perfect best responses. We are interested in a special set of policies:Definition 4 Cooperative Markov policies starting from state s (π DISPLAYFORM2 We let the set of cooperative policies be denoted by Π C i (c). Let the set of policies which are cooperative from any state be the set of perfectly cooperative policies. A social dilemma is a game where there are no cooperative policies which form equilibria. In other words, if one player commits to always cooperate, there is a way for their partner to exploit them and earn higher rewards at their expense. Note that in a social dilemma there may be policies which achieve the payoffs of cooperative policies because they cooperate on the trajectory of play and prevent exploitation by threatening non-cooperation on states which are never reached by the trajectory. The state representation used plays an important role in determining whether equilibria which achieve cooperative payoffs exist. Specifically, a policy which rewards cooperation today with cooperation tomorrow must be able to remember whether cooperation happened yesterday. In both of our example games, Coins and the PPD, if the game is played from the pixels without memory maintaining cooperation is impossible. This is because the current state does not contain information about past behavior of one's partner. Thus, some memory is required to create policies which maintain cooperation. This memory can be learned (eg. an RNN) or it can be an explicitly designed summary statistic (our approach). However, adding memory does not remove equilibria where both players always defect, so adding memory does not imply that self-play will find policies that maintain cooperation BID17 BID47. In the appendix we show that even in the simplest situation, the one memory repeated PD, always defecting equilibria can be more robust attractors than ones which maintain cooperation. amTFT is designed to get around this problem by using modified self-play to explicitly construct the cooperative and cooperation maintaining strategies as well as then switching rule. We begin with the theory behind amTFT. We begin with a social dilemma where pure cooperators can be exploited. We aim to construct a simple meta-policy which incentivizes cooperation along the path of play by switching intelligently between policies in response to its partner. We assume that cooperative polices are exchangeable. That is, for any pair (π DISPLAYFORM0 and that all pairs give a unique distribution of the total rewards between the two players. If policies are not exchangeable or can give different distributions of the total payoff then in addition to having a cooperation problem, we also have a coordination problem (ie. in which particular way should agents cooperate? how should gains from cooperation be split?). This is an important question, especially if we want our agents to interact with humans, and is related to the notion of choosing focal points in coordination/bargaining games. However, a complete solution is beyond the scope of this work and will often depend on contextual factors. See eg. BID48; BID46; BID26; BID42; BID6 for more detailed discussion. For the social dilemma to be solvable, there must be strategies with worse payoffs to both players. Consider an equilibrium (π DISPLAYFORM1 2) which has worse payoffs for player 2. We assume that (π DISPLAYFORM2 is an equilibrium even if played for a finite time, which we call π D -dominance. We use π D -dominance to bound the payoffs of a partner during the execution of a punishment phase, thus it is a sufficient but not necessary condition. We discuss in the Appendix how this assumption can be relaxed. To define this formally, we first introduce the notation of a compound policy π X k Z which is a policy that behaves according to X for k turns and then Z afterwards. Definition 5 We say a game is π D dominant (for player 2) if for any k, any state s, and any policy DISPLAYFORM3 ).In theory, with access to π C, π D, their Q functions, and no noise or function approximation, we can construct amTFT as follows. Suppose the amTFT agent plays as player 1 (the reverse is symmetric).At the start of the game the amTFT agent begins in phase C. If the phase is C then the agent plays according to π C. At each time step, if the agent is in a C phase, the agent looks at the action a 2 chosen by their partner. The agent computes DISPLAYFORM4 If d > 0 then starting at the next time step when state s is reached the agent enters into a D phase where they choose according to π D for k periods. k is computed such that DISPLAYFORM5 Here α > 1 controls how often an agent can be exploited by a pure defector. After this k is over the agent returns to the C phase. The amTFT strategy gives a nice guarantee: DISPLAYFORM6 δ then if player 1 is an amTFT agent, a fully omniscient player 2 maximizes their payoffs by behaving according to π C 2 when 1 is in a C phase and π D 2 when 1 is in a D-phase. Thus, if agents start in the C phase and there is no noise, they cooperate forever. If they start in a D phase, they eventually return to a C phase. The proof is quite simple and we relegate it to the Appendix. However, we now see that amTFT has the desiderata we have asked for: it is easy to explain, it cooperates with a pure cooperator, it does not get completely exploited by a pure defector, 6 and incentivizes cooperation along the trajectory of play. We now use RL methods to construct the components required for amTFT by approximating the cooperative and defect policies as well as the switching policy. To construct the required policies we use self-play and two reward schedules: selfish and cooperative. In the selfish reward schedule each agent i treats the other agent just as a part of their environment and tries to maximize their own reward. We assume that RL training converges and we call the converged policies under the selfish reward scheduleπ In the cooperative reward schedule each agent gets rewards both from their own payoff and the rewards the other agent receives. That is, we modify the reward function so that it is DISPLAYFORM0 6 In an infinite length game amTFT will get exploited an infinite number of times as it tries to return to cooperation after each D phase. One potential way to avoid this to avoid this is to increase α at each D phase. 7 This makes amTFT subtly different from TFT. TFT requires one's partner to cooperate even during the D phase for the system return to cooperation. By contrast, amTFT allows any action during the D phase, this makes it similar to the rPD strategy of Win-Stay-Lose-Shift or Pavlov BID37.We call the converged policy and value function approximationsπ C i andQ i CC. In this paper we are agnostic to which learning algorithm is used to compute policies. In general there can be convergence issues with selfish self-play BID18 BID9 BID40 while in the cooperative reward schedule the standard RL convergence guarantees apply. The latter is because cooperative training is equivalent to one super-agent controlling both players and trying to optimize for a single scalar reward. With the value functions and policies in hand from the procedure above, we can construct an amTFT meta-policy. For the purposes of this construction, we consider agent 1 as the amTFT agent (but everything is symmetric). The amTFT agent keeps a memory state (W t, b t) which both start at 0.The amTFT agent sees the action a of their partner at time t and approximates the gain from this deviation as DISPLAYFORM1 To compute this debit we can either use learned Q functions or we can simply use rollouts. The amTFT agent accumulates the total payoff balance of their partner as W t = W t−1 + D t. If W t is below a fixed threshold T the amTFT agent chooses actions according to π C. If W t crosses a threshold T the mTFT agent uses rollouts to compute a k such that the partner loses more fromπ DISPLAYFORM2 relative to cooperation than some constant α times the current debit. The hyperparameters T and α trade off robustness to approximation error and noise. Raising T allows for more approximation error in the calculation of the debit but relaxes the incentive constraints on the agent's partner. Raising α makes the cost of defection higher but makes false positives more costly. The algorithm is formalized below:Algorithm 1 Approximate Markov Tit For Tat (for Agent 1) 2 ) -we call this the'counterfactual path' (c) The amTFT agent takes the difference in the average total reward to the partner from the two paths and uses that as D t -this is an estimate of the reward of the one shot deviation to a from the recommended strategyπ DISPLAYFORM3 DISPLAYFORM4 This procedure is an unbiased estimator of Q CC in the limit of large B and M but is computationally intensive at test time. 8 In games where an action today can only affect payoffs up to M periods from now it suffices to use rollouts of length M and elide the continuation value. The value-based construction gives amTFT a particular robustness property -if the partner is not usingπ C 2 exactly but is using a policy that is outcome equivalent to it the estimated D t values will end up being 0 in expectation and so the amTFT agent will continue to cooperate. We will see in our experiments that this property is important to the success of amTFT in real Markov social dilemmas. We test amTFT in two environments: one grid-world and one where agents must learn from raw pixels. In the grid-world game Coins two players move on a 5 × 5 board. The game has a small probability of ending in every time step, we set this so the average game length is 500 time steps. Coins of different colors appear on the board periodically, and a player receives a reward of 1 for collecting (moving over) any coin. However, if a player picks up a coin of the other player's color, the other player loses 2 points. The payoff for each agent at the end of each game is just their own point total. The strategy which maximizes total payoff is for each player to only pick up coins of their own color; however each player is tempted to pick up the coins of the other player's color. We also look at an environment where strategies must be learned from raw pixels. We use the method of to alter the reward structure of Atari Pong so that whenever an agent scores a point they receive a reward of 1 and the other player receives −2. We refer to this game as the Pong Player's Dilemma (PPD). In the PPD the only (jointly) winning move is not to play. However, a fully cooperative agent can be exploited by a defector. We are interested in constructing general strategies which scale beyond tabular games so we use deep neural networks for state representation for both setups. We use standard setups so we relegate the details of the networks as well as the training to the appendix. We perform both Selfish (self play with reactive agents receiving own rewards) and Cooperative (self play with both agents receiving sum of rewards) training for both games. We train 100 replicates for Coins and 18 replicates for the PPD. In both games Selfish training leads to suboptimal behavior while Cooperative training does find policies that implement socially optimal outcomes. In Coinŝ π D agents converge to picking up coins of all colors while socialπ C agents learn to only pick up matching coins. In PPD selfishly trained agents learn to compete and try to score while prosocially trained agents gently hit the ball back and forth. In two Markov social dilemmas we find that standard self-play converges to defecting strategies while modified self-play finds cooperative, but exploitable strategies. We use the of these two training schedules to constructπ C andπ D.8 A less computationally demanding way to execute amTFT is to use a model to approximate QCC directly. This is difficult in practice since any bias in the model is accumulated across periods and because the model needs to be accurate everywhere, not just on the trajectory of π C. In the appendix we discuss some on learning a modelQ and improving the efficiency of such procedures is an important direction for future work. Figure 2: In two Markov social dilemmas, amTFT satisfies the Axelrod desiderata: it mostly cooperates with itself, is robust against defectors, and incentivizes cooperation from its partner. The'Grim' strategy based on BID4 behaves almost identically to pure defection in these social dilemmas. The of standard self-play is π D. The full tournament of all strategies against each other is shown in the Appendix. We evaluate the performance of various Markov social dilemma strategies in a tournament. To construct a matchup between two strategies we construct agents and have them play a fixed length iteration of the game. Note that at training time we use a random length game but at test time we use a fixed length one so that we can compare payoffs more efficiently. We use 1000 replicates per strategy pair to compute the average expected payoff. We compareπ C,π D, and amTFT.We also compare the direct adaptation of the construction in BID4. Recall that the folk theorem algorithm maintains equilibria by threat of deviation later: if either agent's behavior in game iteration t does not accord with the cooperative policy, both agents switch to a different policy in the next repetition of the game. We adapt this to the single test game setting as follows: the agent computes policiesπ C,π D. If their partner j takes an action a in a state s where a =π C j (s) the agent switches toπ D forever. We call this the Grim Trigger Strategy due to its resemblance to the rPD strategy of the same name. In both games we ask how well the strategies satisfy Axelrod's desiderata from the introduction. Specifically, we would like to measure whether a strategy avoids exploitation, cooperates with conditional cooperators, and incentivizes its partner to cooperate. Let S i (X, Y) be the average reward to player i when a policy of type X is matched with type Y. The metric DISPLAYFORM0 measures how safe a strategy is from exploitation by a defector. The larger this value, the worse that π X is exploited by a pure defector. We measure a strategy's ability to achieve cooperative outcomes with policies of their same type as DISPLAYFORM1 This measure can be thought of as quantifying two things. First, how much social welfare is achieved in a world where everyone behaves according to strategy X. Second, while we cannot enumerate all possible conditionally cooperative strategies, in the case of Grim and amTFT this serves as an indicator of how well they would behave against a particular conditional cooperator -themselves. Finally, we measure if X incentivizes cooperation from its partner. For this we use the measure DISPLAYFORM2 The higher this number, the better off a partner is from committing to pure cooperation rather than trying to cheat. Figure 2 shows our metrics evaluated for the strategies of always cooperate, always defect, amTFT and Grim. Pure cooperation is fully exploitable and pure defection gets poor payoffs when matched with itself. Neither pure strategy incentivizes cooperation. amTFT avoids being exploited by defectors, does well when paired with itself and incentivizes cooperative strategies from its partner. We also see that inferring a partner's cooperation using the value function (amTFT) is much more stable than inferring it via actions (Grim). The above show that amTFT is a good strategy to employ in a mixed environment which includes some cooperators, some tit-for-tat agents and some defectors. We consider what happens if we fix the one player (the Teacher) to use a fixed policy but let the other player be a selfish deep RL agent (the Learner). We perform the retraining in the domain of Coins. 9 This retraining procedure can also be used as an additional metric of the exploitability of a given strategy, rather than asking whetherπ D can exploit it, we ask whether a learner trying to maximize its own payoff can find some way to cheat. Recall that when selfish RL agents played with each other, they converged to the Selfish'grab all coins' strategy. We see that Learners paired with purely cooperative teachers learn to exploit the teachers, learners paired withπ D also learn to exploit (this learning happens much slower because a fully trainedπ D policy is able to grab coins very quickly and thus it is hard for a blank slate agent to learn at all), however learners paired with amTFT learn to cooperate. Note that choosing amTFT as a strategy leads to higher payoffs for both the Learner and the Teacher, thus even if we only care about the payoffs accrued to our own agent we can do better with amTFT than a purely greedy strategy. Humans are remarkably adapted to solving bilateral social dilemmas. We have focused on how to give artificial agents this capability. We have shown that amTFT can maintain cooperation and avoid exploitation in Markov games. In addition we have provided a simple construction for this strategy that requires no more than modified self-play. Thus, amTFT can be applied to social dilemmas in many environments. Our emphasize the importance of treating agents as fundamentally different than other parts of the environment. In particular, agents have beliefs, desires, learn, and use some form of optimization while objects follow simple fixed rules. An important future direction for constructing cooperative agents is to continue to incorporate ideas from inverse reinforcement learning BID0 BID36 and cognitive science BID5 BID26 to construct agents that exhibit some theory of mind. There is a growing literature on hybrid systems which include both human and artificial agents BID10 BID50. In this work we have focused on defining'cooperation' as maximizing the joint payoff. This assumption seems reasonable in symmetric situations such as those we have considered, however, as we discuss in the introduction it may not always be appropriate. The amTFT construction can be easily modified to allow other types of focal points simply by changing the modified reward function used in the training of the cooperative strategies (for example by using the inequity averse utility functions of BID15). However moving forward in constructing agents that can interact in social dilemmas with humans will require AI designers (and their agents) to understand and adapt to human cooperative and moral intutions BID27; BID21 BID39 In a social dilemma there exists an equilibrium of mutual defection, and there may exist additional equilibria of conditional cooperation. Standard self-play may converge to any of these equilibria. When policy spaces are large, it is often the case that simple equilibria of constant mutual defection have larger basins of attraction than policies which maintain cooperation. We can illustrate this with the simple example of the repeated Prisoner's Dilemma. Consider a PD with payoffs of 0 to mutual defection, 1 for mutual cooperation, w > 1 for defecting on a cooperative partner and −s for being defected on while cooperating. Consider the simplest possible state representation where the set of states is the pair of actions played last period and let the initial state be (C, C) (this is the most optimistic possible setup). We consider RL agents that use policy gradient ( displayed here come from using Adam BID25, similar were obtained with SGD though convergence speed was much more sensitive to the setting of the learning rate) to learn policies from states (last period actions) to behavior. Note that this policy space contains TFT (cooperate after (C, C), (D, C), defect otherwise), Grim Trigger (cooperate after (C, C), defect otherwise) and Pavlov or Win-Stay-Lose-Shift (cooperate after (C, C), (D, D), defect otherwise BID37 ) which are all cooperation maintaining strategies (though only Grim and WSLS are themselves full equilibria).Each episode is defined as one repeated PD game which lasts a random number of periods with stopping probability of stopping.05 after each period. Policies in the game are maps from the onememory state space {(C, C), (D, C), (C, D), (D, D)} to either cooperation or not. These policies are trained using policy gradient and the REINFORCE algorithm . We vary w and set s = 1.5w such that (C, C) is the most efficient strategy always. Note that all of these parameters are well within the range where humans discover cooperative strategies in experimental applications of the repeated PD BID7. Figure 4 shows that cooperation only robustly occurs when it is a dominant strategy for both players (w < 0) and thus the game is no longer a social dilemma. 10.10 Note that these use pairwise learning and therefore are different from evolutionary game theoretic on the emergence of cooperation BID38. Those show that indeed cooperation can robustly emerge in these kinds of strategy spaces under evolutionary processes. Those differ because they rely on the following argument: suppose we have a population of defectors. This can be invaded by mutants of TFT because TFT can try cooperation in the first round. If it is matched with a defector, it loses once but it then defects for the rest of the time, if it is matched with another TFT then they cooperate for a long time. Thus, for Figure 4: Results from training one-memory strategies using policy gradient in the repeated Prisoner's Dilemma. Even in extremely favorable conditions self-play fails to discover cooperation maintaining strategies. Note that temptation payoff.5 is not a PD and here C is a dominant strategy in the stage game. To prove the theorem we will apply the one deviation principle. To show this, we fix player 1 to be an amTFT agent and look at player 2. Note that from the point of view of player 2 this is now a Markov game with a state representation of (s, k) where if k = 0 player 1 behaves according to π C and if k > 0 player 1 is in the D phase and thus behaves according to π D k C.We consider the policy for player 2 of'play π C 2 when player 1 is in the C phase and play π D 2 when player 1 is in the D phase.' Recall by the Principle of Optimality if there does not exist a one shot deviation a at any state under which player 2 earns a higher payoff, then there does not exist a better policy than the one prescribed. Consider starting at k > 0. The suggested policy has player 2 play π DISPLAYFORM0. By π D -dominance this is the best response to π DISPLAYFORM1 so there are no one-shot deviations in the D phase. Let us consider what happens in the C phase (k = 0). By the assumption of the theorem at any state s we know that DISPLAYFORM2 Let {r t 2 (s, π 1, π 2)} be the per-period reward stream (note here each r is a random variable) for player 2 induced by the policies π 1, π 2. Since DISPLAYFORM3 where δ is the discount rate. Because rewards are bounded then for any > 0 there exists k such that DISPLAYFORM4 sufficiently long games the risk of one round of loss is far smaller than the potential fitness gain of meeting another mutant. Thus TFT can eventually gain a foothold. It is clear why in learning scenarios such arguments cannot apply. That is, the first k terms time steps approximate the full discounted expectation arbitrarily well. This also means that for some k DISPLAYFORM5 From any state, the highest profit an agent can make from deviating from π C 2 with a single action with an amTFT partner is d *. However we have shown that there exists a length k such that moving to D for k turns costs the agent more than d * δ. Therefore there is no time during the C phase they wish to deviate. This completes the proof. DISPLAYFORM6 The reason we made the π D -dominance assumption is to bound the expected payoff of an agent playing against π D k C and therefore bound the necessary length of a D phase after a particular deviation. However, in order to compute what the length of the D phase the amTFT agent needs access to the best response policy to π D k C, or its associated value function. With π D -dominance we assume that π D is that best response. Even if π D -dominance does not strictly hold, it is likely a sufficient approximation. If necessary however, one can train an RL agent on episodes where their partner plays π D k C, where k is observed. This allows one to approximate the best response policy to π D k C which will then give us what we need to compute the responses to deviations from π C in the D phase that incentivize full cooperation. We used rollouts to calculate the debit to the amTFT's partner at each time period. This estimator has good performance for both PPD and Coins given their reward structure. It is also possible to use a learned model of Q. Learning a sufficiently accurate modelQ is challenging for several reasons. First, it has to have very low bias, since any bias inQ will be accumulated over periods. Second, the one-shot deviation principle demands thatQ be accurate for all state-action pairs, not just those sampled by the policies (π C, π C). Standard on-policy value function estimation will only produce accurate estimates of Q at states sampled by the cooperative policies. As an example, in Coins, since the cooperative policies never collect their partner's coinsQ for these state-action pairs may be inaccurate. We found that it was possible in Coins to learn a modelQ to calculate debit without policy rollouts using the same neural network architecture that was used to train the policies. However, we found that in order to train aQ model accurate enough to work well we had to use a modified training procedure. After finishing Selfish and Cooperative training, we perform a second step of training using a fixed (converged)π C. In order to sample states off the path ofπ C during this step, the learner behaves according to a mixture of π C, π D, and random policies while the partner continues according tô π C.Q is updated via off-policy Bellman iteration. We found this modified procedure produced â Q function that was good enough to maintain cooperation (though still not as efficient as rollouts). For more complex games, an important area for future work is to develop methodologies to compute more accurate approximations of Q or combine aQ model with rollouts effectively. For Coins there are four actions (up, down, left, right), and S is represented as a 4 × 5 × 5 binary tensor where the first two channels encode the location of the each agent and the other two channels encode the location of the coin (if any exist). At each time step if there is no coin on the board a coin is generated at a random location with a random color, with probability 0.1.A policy π(s; θ): s → ∆(a) is learned via the advantage actor critic algorithm. We use a multi-layer convolutional neural network to jointly approximate the policy π and state-value functionV. For this small game, a simpler model could be used, but this model generalizes directly to games with higher-dimensional 2D state spaces (e.g. environments with obstacles). For a given board size k, the model has log 2 (k) + 1 repeated layers, each consisting of a 2D convolution with kernel size 3, followed by batch normalization and ReLU. The first layer has stride 1, while the successive layers each have stride 2, which decreases the width and height from k to k/2 while doubling the number of channels. For the 5 × 5 board, channel sizes are 13, 26, 52, 104. From these 104 features, π is computed via a linear layer with 4 outputs with softmax, to compute a distribution over actions, while the value function is computed via a single-output linear layer. The actor and critic are updated episodically with a common learning rate -at the end of each game we update the model on a batch of episodes via ∆θ i = λ A t ∂V (s t) ∂θ i +Ã t log π(s t, a t) ∂π(s t, a t) ∂θ i where A is the advantage A t = r t + δV (s t+1) − V (s t)andÃ is the advantaged normalized over all episodes and periods in the batch DISPLAYFORM0 We train with a learning rate of 0.001, continuation probability.998 (i.e. games last on average 500 steps), discount rate 0.98, and a batch size of 32. We train for a total of 40, 000 games. We use the arcade learning environment modified for 2-player play as proposed in , with modified rewards of +1 for scoring a point and -2 for being scored on. We train policies directly from pixels, using the pytorch-a3c package https://github.com/ikostrikov/ pytorch-a3c. Policies are trained directly from pixels via A3C BID35. Inputs are rescaled to 42x42 and normalized, and we augment the state with the difference between successive frames with a frame skip of 8. We use 38 threads for A3C, over a total of 38,000 games (1,000 per thread). We use the default settings from pytorch-a3c: a discount rate of 0.99, learning rate of 0.0001, 20-step returns, and entropy regularization weight of 0.01.The policy is implemented as a convolutional neural network with four layers, following pytorch-a3c. Each layer uses a 3x3 kernel with stride 2, followed by ELU. The network has two heads for the actor and critic. We elide the LSTM layer used in the pytorch-a3c library, as we found it to be unnecessary. (a) Coins Results (b) PPD Results Figure 5: Results of the tournament in two Markov social dilemmas. Each cell contains the average total reward of the row strategy against the column strategy. amTFT achieves close to cooperative payoffs with itself and achieves close to the defect payoff against defectors. Its partner also receives a higher payoff for cooperation than defection.

Represent the TLDR for finding the corresponding scientific article:

How can we build artificial agents that solve social dilemmas (situations where individuals face a temptation to increase their payoffs at a cost to total welfare)?

We use reinforcement learning for query reformulation on two tasks and surprisingly find that when training multiple agents diversity of the reformulations is more important than specialisation.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: The reparameterization trick has become one of the most useful tools in the field of variational inference. However, the reparameterization trick is based on the standardization transformation which restricts the scope of application of this method to distributions that have tractable inverse cumulative distribution functions or are expressible as deterministic transformations of such distributions. In this paper, we generalized the reparameterization trick by allowing a general transformation. Unlike other similar works, we develop the generalized transformation-based gradient model formally and rigorously. We discover that the proposed model is a special case of control variate indicating that the proposed model can combine the advantages of CV and generalized reparameterization. Based on the proposed gradient model, we propose a new polynomial-based gradient estimator which has better theoretical performance than the reparameterization trick under certain condition and can be applied to a larger class of variational distributions. In studies of synthetic and real data, we show that our proposed gradient estimator has a significantly lower gradient variance than other state-of-the-art methods thus enabling a faster inference procedure. Most machine learning objective function can be rewritten in the form of an expectation: where θ is a parameter vector. However, due to the intractability of the expectation, it's often impossible or too expensive to calculate the exact gradient w.r.t θ, therefore it's inevitable to estimate the gradient ∇ θ L in practical applications. Stochastic optmization methods such as reparameterization trick and score function methods have been widely applied to address the stochastic gradient estimation problem. Many recent advances in large-scale machine learning tasks have been brought by these stochastic optimization tricks. Like in other stochastic optimzation related works, our paper mainly focus on variational inference tasks. The primary goal of variational inference (VI) task is to approximate the posterior distribution in probabilistic models . To approximate the intractable posterior p(z|x) with the joint probability distribution p(x, z) over observed data x and latent random variables z given, VI introduces a parameteric family of distribution q θ (z) and find the best parameter θ by optimizing the Kullback-Leibler (KL) divergence D KL (q(z; θ) p(z|x)). The performance of VI methods depends on the capacity of the parameteric family of distributions (often measured by Rademacher complexity) and the ability of the optimizer. In this paper, our method tries to introduce a better optimizer for a larger class of parameteric family of distributions. The main idea of our work is to replace the parameter-independent transformation in reparameterization trick with generalized transformation and construct the generalized transformation-based (G-TRANS) gradient with the velocity field which is related to the characteristic curve of the sublinear partial differential equation associated with the generalized transformation. Our gradient model further generalizes the G-REP and provides a more elegant and flexible way to construct gradient estimators. We mainly make the following contributions: 1. We develop a generalized transformation-based gradient model based on the velocity field related to the generalized transformation and explicitly propose the unbiasedness constraint on the G-TRANS gradient. The proposed gradient model provides a more poweful and flexible way to construct gradient estimators. 2. We show that our model is a generalization of the score function method and the reparameterization trick. Our gradient model can reduce to the reparameterization trick by enforcing a transport equation constraint on the velocity field. We also show our model's connection to control variate method. 3. We propose a polynomial-based gradient estimator that cannot be induced by any other existing generalized reparameterization gradient framework, and show its superiority over similar works on several experiments. The rest of this paper is organized as follows. In Sec.2 we review the stochastic gradient variational inference (SGVI) and stochastic gradient estimators. In Sec.3 we propose the generalized transformation-based gradient. In Sec.4 we propose the polynomial-based G-TRANS gradient estimator. In Sec.5 we study the performance of our gradient estimator on synthetic and real data. In Sec.6 we review the related works. In Sec.7 we conclude this paper and discuss future work. To obtain the best variational parameter θ, rather than minimize the KL divergence D KL (q(z; θ) p(z|x)), we usually choose to maximize the evidence lower bound (ELBO) , where The entropy term H[q(z; θ)] is often assumed to be available analytically and usually omitted in the procedure of stochastic optimization. This stochastic optimization problem is the basic setting for our method and experiments. Without extra description, we only consider the simplified version of the ELBO: Generally, this expectation is intractable to compute, let alone its gradient. Therefore, a common stochastic optimization method for VI task is to construct a Monte Carlo estimator for the exact gradient of the ELBO w.r.t θ. Among those gradient estimators, the score function method and the reparamterization trick are most popular and widely applied. Score function method. The score function estimator, also called log-derivative trick or reinforce; is a general way to obtain unbiased stochastic gradients of the ELBO (; ;). The simplest variant of the score function gradient estimator is defined as: and then we can build the Monte Carlo estimator by drawing samples from the variational distribution q θ (z) independently. Although the score function method is very general, the ing gradient estimator suffers from high variance. Therefore, it's necessary to apply variance reduction (VR) methods such as RaoBlackwellization and control variates in practice. Reparameterization trick. In reparameterization trick, we assume that there is an invertible and continuously differentiable standardization function φ(z, θ) that can transform the variational distribution q(z; θ) into a distribution s(ρ) that don't depend on the variational parameter θ as follows, θ (ρ) Then the reparameterization trick can turn the computation of the gradient of the expectation into the expectation of the gradient: Although this reparameterization can be done for many commonly used distributions, such as the Gaussian distribution, it's hard to find appropriate standardization functions for a number of standard distributions, such as Gamma, Beta or Dirichlet because the standardization functions will inevitably involve special functions. On the other hand, though the reparameterization trick is not as generally applicable as the score function method, it does in a gradient estimator with lower variance. Define a random variable ρ by an invertible differentiable transformation ρ = φ(z, θ), where φ is commonly called generalized standardization transformation since it's dependent on the variational parameter θ. Theorem 3.1. Let θ be any component of the variational parameter θ, the probability density function of ρ be w(ρ, θ) and where The proof details of the Theorem.3.1 are included in the Appendix. A.1. We refer to the gradient ∂L ∂θ with v θ satisfying the unbiasedness constraint as generalized transformation-based (G-TRANS) gradient. We can construct the G-TRANS gradient estimator by choosing v θ of specific form. In the following, we demonstrate that the score function gradient and reparameterization gradient are special cases of our G-TRANS gradient model associating with special velocity fields. Remark. The score function method is a special case of the G-TRANS model when The standardization function φ doesn't depend on the parameter θ when v θ = 0 according to the velocity field equation (Equ.5). Conversely, for any φ that doesn't depend on θ, we have = 0, thus the ing gradient estimator has a same variance as the score function estimator. Remark. The reparameterization trick is a special case when The detailed computation to obtain the transport equation (Equ.7) is included in the Appendix. A.1. The transport equation is firstly introduced by , however, their work derive this equation by an analog to the optimal transport theory. In 1-dimensional case, for any standardization distributions w(ρ) that doesn't depend on the parameter θ, the variance of the ing gradient estimator is some constant (for fixed θ) determined by the unique 1-dimensional solution of the transport equation. For the existence of the velocity field v θ and the generalized standardization transformation φ(z, θ), g(z, θ) must satisfy some strong differential constraints . We can see that the G-TRANS model is a special case of the control variate method with a complex differential structure. This connection to CV means our gradient model can combine the advantages of CV and generalized reparameterization. Theorem.3.1 transforms the generalized unbiased reparameterization procedure into finding the appropriate velocity field that satisfy the unbiasedness constraint. It's possible to apply variational optimization theory to find the velocity field with the least estimate variance, however, the solution to the Euler-Lagrange equation contains f (z) in the integrand which makes it impractical to use in real-world model (See Appendix. A.2 for details). By introducing the notion of velocity field, we provide a more elegant and flexible way to construct gradient estimator without the need to compute the Jacobian matrix for a specific transformation. In the next section, we introduce a polynomial-based G-TRANS gradient estimator that cannot be incorporated into any other existing generalized reparameterized gradient framework and is better than the reparameterization gradient estimator theoretically. In this section, we always assume that the base distribution q(z, θ) can be factorized as where N is the dimension of the random variable z, θ i is a slice of θ and θ i share no component with θ j if i = j. We consider an ad-hoc velocity field family: We always assume v θ ah to be continuous which guarantees the existence of the solution to the velocity field equation. We verify in the Appendix. A.3 that v θ ah (z, θ) satisfy the unbiasedness constraint if h(z, θ) is bounded. It's easy to see that the gradient estimator that from v θ ah is more general than the score function method or reparameterization trick since they are two special cases when h(z, θ) = 0 or h(z, θ) = f (z) respectively. In this paper, we mainly consider a more special family of the v θ ah (z, θ): where zi ∂q(z,θ) ∂θ dz i ), but their properties are similar (we present some theoretical of v θ dp in the Appendix. A.4). Therefore we only consider v θ poly (z, θ) here. We refer to v θ poly as polynomial velocity field. Proposition 4.1. For distributions with analytical high order moments such as Gamma, Beta or Dirichlet distribution, the expectation are polynomials of random variable z. Therefore, for distribution with analytical high order moments, With Proposition.4.1, we can write the G-TRANS gradient for the polynomial velocity field as: Thus we can construct a G-TRANS gradient estimator based upon the polynomial velocity field with a samplez drawn from q(z, θ): The polynomial-based G-TRANS gradient estimator has a form close to control variate, thus cannot be induced by any other existing generalized reparameterized gradient framework. In the following, we show that the polynomial-based G-TRANS gradient estimator performs better than the reparameterization gradient estimator under some condition. dz N ), then the gradient estimator ed from polynomial velocity field has a smaller variance than the reparameterization gradient estimator. Proof. Since E q [P k (z, θ) ∂ ∂θ log q] can be resolved analytically, we have then by reorganizing the expression Var(− ∂f ∂zi), we can prove this proposition. As an example about how to choose a good polynomial, for ), we can obtain a polynomial-based G-TRANS gradient estimator that is better than the reparameterization gradient estimator according to the Proposition.4.2. And we can adjust the value of C i (θ) to obtain better performance. According to the approximation theory, we can always find a polynomial P k (z, θ) that is close enough to f (z), and in this case, we can dramatically reduce the variance of the ing gradient estimator. For example, within the convergence radius, we can choose P k (z, θ) to be the k-th degree Taylor polynomial of f (z) with the remainder |f (z) − P k (z, θ)| being small. In the practical situation, however, it's often difficult to estimate the coefficients of the polynomial P k (z, θ). And when k is large, we need to estimate O(N k) coefficients which is almost impossible in real-world applications. Therefore in the following experiments, we only consider k < 2. In this section, we use a Dirichlet distribution to approximate the posterior distribution for a probilistic model which has a multinomial likelihood with a Dirichlet prior. We use Gamma distributions to simulate Dirichlet distributions. If Then the problem we study here can be written as: with f (z) being the multinomial log-likelihood. We use shape parameter α = (α 1, . . ., α K) to parameterize the variational Dirichlet distribution. To construct polynomial-based G-TRANS gradient estimator for the factorized distribution K k=1 Gamma(z k ; α k, 1), we need an accurate and fast way to approximate the derivative of the lower incomplete gamma function (part of the gamma CDF) w.r.t the shape parameter. The lower incomplete gamma function γ(α, z) is a special function and does not admit analytical expression for derivative w.r.t. the shape parameter. However, for small α and z, we have In practice, we take the first 200 terms from this power series. And the approximation error is smaller than 10 −9 when α < 5 and z < 20 with double precision floating point number. For large α, we use central finite difference to approximate the derivative. This approximation scheme for lower incomplete gamma function can also be used to construct polynomial-based G-TRANS gradient estimator for distributions that can be simulated by the Gamma distribution such as Beta distribution and Dirichlet distribution. We follow the experiment setting in. Fig.1 shows the ing variance of the first component of the gradient based on samples simulated from a Dirichlet distribution with K = 100 components, and gradients are computed with N = 100 trials. We use P 1 (z) = c · z to construct the G-TRANS gradient estimator, and we assign 0.2,0 and −0.1 to c successively as α 1 increases. Results. From Fig.1, we can see that the IRG method and our G-TRANS gradient estimator has obviously lower gradient variance than the RSVI (even with the shape augmentation trick ) or G-REP method. Further, our G-TRANS gradient estimator outperforms the IRG method when α 1 is large though there is no obvious difference between these two methods when α 1 is small. In this section, we study the performance of our G-TRANS gradient estimator on the Sparse Gamma deep exponential family (DEF) model with the Olivetti faces dataset that consists of 64 × 64 gray-scale images of human faces in 8 bits. We follow the Sparse Gamma DEF setting in where the DEF model is specified by: . C is the polynomial coefficient, B denotes shape augmentation and optimal concentration is α = 2. Here n is the number of observations, is the layer number, k denotes the k-th component in a specific layer and d is the dimension of the output layer (layer 0). z n,k is local random variable, w k,k is global weight that connects different layers like deep neural networks, and x n,d denotes the set of observations. We use the experiment setting in. α z is set to 0.1, all priors on the weights are set to Gamma(0.1, 0.3), and the top-layer local variables priors are set to Gamma(0.1, 0.1). The model consists of 3 layers, with 100, 40, and 15 components in each. All variational Gamma distributions are parameterized by the shape and mean. For non-negative variational parameters θ, the transfomration θ = log(1 + exp(ϑ)) is applied to avoid constrained optimization. In this experiment, we use the step-size sequence ρ n proposed by: δ = 10 −16, t = 0.1, η = 0.75 is used in this experiment. The best of RSVI is reproduced with B = 4 . We still use P 1 (z) = c · z to construct the G-TRANS gradient estimator and we use c = −10.0 for all time. Results. From Fig.2, We can see that G-TRANS achieves significant improvements in the first 1000 runs and exceeds RSVI though with a slower initial improvement. G-TRANS achieves obviously better accuracy than ADVI, BBVI, G-REP and RSVI, and keeps improving the ELBO even after 75000 runs. G-TRANS is faster than the IRG in early training stage which means G-TRANS has a lower gradient variance. However, this speed advantage of G-TRANS gradually decreases as the step size goes down in the later training stage. There are already some lines of research focusing on extending the reparameterization trick to a larger class of distributions. The G-REP generalizes the reparameterization gradient by using a standardization transformation that allows the standardization distribution to depend weakly on variational parameters. Our gradient model gives a more elegant expression of the generalized reparameterized gradient than that of G-REP which decomposes the gradient as g rep + g cor. Different from G-REP, our model hides the transformation behind the velocity field thus the expensive computation of the Jacobian matrix of the transformation is evaded. And it's more flexible to construct gradient estimator with the velocity field than the very detailed transformation. The RSVI develops a similar generalized reparameterized gradient model with the tools from rejection sampling literatures. RSVI introduces a score function gradient term to compensate the gap that is caused by employing the proposal distribution of a rejection sampler as a surrogate distribution for reparameterization gradient, although the score function gradient term can often be ignored in practice to reduce the gradient variance at the cost of small bias. Unlike RSVI, our gradient estimator can be constructed with deterministic procedure which avoids the additional stochasticity introduced by the accept-reject steps thus lower gradient variance. The path-wise derivative is closely related to our model. They obtain the transport equation by an analog to the displacement of particles, while we derive the transport euqation for reparameterization gradient by rigorous mathematical deduction. The path-wise gradient model can be seen as a special case of our G-TRANS gradient model. Their work only focus on standard reparameterization gradient while our model can admit generalized transformation-based gradient. The velocity field used in their work must conform to the transport equation while we only require the velocity field to satisfy the unbiasedness constraint. The implicit reparameterization gradient (IRG) differentiates from the path-wise derivative only by adopting a different method for multivariate distributions. There are also some other works trying to address the limitations of standard reparameterization. applies implicit reparameterization for mixture distributions and uses approximations to the inverse CDF to derive gradient estimators. Both work involve expensive computation that cannot be extended to large-scale variational inference. expressed the gradient in a similar way to G-REP and automatically estimate the gradient in the context of stochastic computation graphs, but their work is short of necessary details therefore cannot be applied to general variational inference task directly. ADVI transforms the random variables such that their support are on the reals and then approximates transformed random variables with Gaussian variational posteriors. However, ADVI struggles to approximate probability densities with singularities as noted by. We proposed a generalized transformation-based (G-TRANS) gradient model which extends the reparameterization trick to a larger class of variational distributions. Our gradient model hides the details of transformation by introducing the velocity field and provides a flexible way to construct gradient estimators. Based on the proposed gradient model, we introduced a polynomial-based G-TRANS gradient estimator that cannot be induced by any other existing generalized reparameterization gradient framework. In practice, our gradient estimator provides a lower gradient variance than other state-of-the-art methods, leading to a fast converging process. For future work, We can consider how to construct G-TRANS gradient estimators for distributions that don't have analytical high-order moments. We can also utilize the from the approximation theory to find certain kinds of high-order polynomial functions that can approximate the test function effectively with cheap computations for the coefficients. Constructing velocity fields with the optimal transport theory is also a promising direction. A.1 PROOF OF THEOREM.3.1 We assume that transformed random variable ρ = φ(z, θ) is of the same dimension as z. And we assume that there exists ψ(ρ, θ) that satisfy the constraint z = ψ(φ(z, θ), θ). Firstly, by the change-of-variable technique, we have Take derivative w.r.t θ (any component of θ) at both sizes, we have With the rule of determinant derivation, we have Substitute the Equ.19 into Equ.18, we have, we obtain the first of the Theorem.3.1. As for the second part, we have Thus we obtain the second part of the Theorem.3.1. Proof ends. As a by-product, if we make ∂ ∂θ w(φ(z, θ), θ) = 0, we can obtain the transport equation for the reparameterization trick: And ∂ ∂θ w(φ(z, θ), θ) = 0 also means that the standardization distribution is independent with θ which is the core of the reparameterization trick. For the simplicity of the proof, we only consider the 1-dimensional here. And denote where with the unbiased constraint, we have E q(z,θ) [r(z, θ)] = E q(z,θ) [f (z) ∂q(z,θ) ∂θ q(z,θ) ] = const, so we need to consider the term E q(z,θ) [(r θ (z, θ)) 2 ] only. According to the Euler-Lagrange equation, we have Simplify it, we have (f ∂q ∂θ Then we have Thus we have which is usually intractable in real world practice. Here we verify that v If h(z, θ) is bounded, we have Therefore, E q θ [If we take the dual polynomial velocity field v θ dp in the G-TRANS framework, we can reach a dual to the Proposition.4.2: Proposition A.1. If Cov(P k ∂ log q(z,θ) ∂θ, (2f − P k) ∂ log q(z,θ) ∂θ ) > 0, then the gradient estimator ed from dual polynomial velocity field has a smaller gradient variance than the score function gradient estimator. The proof is similar to that of Proposition.4.2.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a novel generalized transformation-based gradient model and propose a polynomial-based gradient estimator based upon the model.

Represent the TLDR for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the scientific article to find a short TLDR.

Question: The fault diagnosis in a modern communication system is traditionally supposed to be difficult, or even impractical for a purely data-driven machine learning approach, for it is a humanmade system of intensive knowledge. A few labeled raw packet streams extracted from fault archive can hardly be sufficient to deduce the intricate logic of underlying protocols. In this paper, we supplement these limited samples with two inexhaustible data sources: the unlabeled records probed from a system in service, and the labeled data simulated in an emulation environment. To transfer their inherent knowledge to the target domain, we construct a directed information flow graph, whose nodes are neural network components consisting of two generators, three discriminators and one classifier, and whose every forward path represents a pair of adversarial optimization goals, in accord with the semi-supervised and transfer learning demands. The multi-headed network can be trained in an alternative approach, at each iteration of which we select one target to update the weights along the path upstream, and refresh the residual layer-wisely to all outputs downstream. The actual show that it can achieve comparable accuracy on classifying Transmission Control Protocol (TCP) streams without deliberate expert features. The solution has relieved operation engineers from massive works of understanding and maintaining rules, and provided a quick solution independent of specific protocols. A telecommunications network is a collection of distributed devices, entirely designed and manufactured by humans for a variety of transmission, control and management tasks, striving to provide a transparent channel between external terminals, via an actual internal relay process node by node. As a typical conversation in the style of client and server, the two linked nodes send their messages in the form of packets, encapsulated the load with miscellaneous attributes in headers to ensure the correctness, consistency, and smoothness of the entire process. A typical header includes packet sequence number, source and destination addresses, control bits, error detection codes, etc. The large-scale network cannot always work ideally, due to its inherent complexity inside massive devices and their interactions. When there is a malfunction of a device, either caused by the traffic overload, or software bugs, or hardware misconfiguration, or malicious attacks, it will be reflected on the packet streams that pass through, such as packet loss, timeout, out of order, etc. System administrators captured those suspicious streams and sent back to the service center for cautious offline analysis, which is time-consuming and domain-specific. The primary challenge of automatic diagnosis is that, it is almost impossible to formalize all the logic inside the system and make them available to artificial intelligence. A typical modern communication system consists of tens of thousands devices end-to-end and runs based on a list of hundreds of protocols layer-by-layer BID6 ). If we could figure out the latent states of protocols by constructing specific features from raw bytes, the subsequent classification tasks would be quite straightforward and easy to implement. For instance, the Transmission Control Protocol (TCP) relies on sequence numbers to judge the receiving order of packets, which may be just big integers roughly linearly growing from the view of machine learning models. Another example is a few critical control bits may reside among much more useless bits, such as checksum codes, which is harmful noises for models. Even we have the patience to dive into all the industrial protocols and build up an exhausted feature library; eventually, we will fail again to achieve the target of automation, one of the main advantages of the modern data-driven approach. Another difficulty is scarce of labeled samples. In spite of there are seemingly numerous packet flows running through the Internet all the time, the real valid faults occur at random and occupy only a tiny portion of whole traffic volume. The actual labeled data are usually collected from the archive of fault cases, which is hard to have enough samples for all possible categories, or cannot at least cover them completely. The previous works on this issue mainly follow two technical routes: 1) a traditional two-phase framework, using expert features and some general-propose classifiers BID1 ); 2) an end-to-end approach based on deep learning for automatic feature extraction . All these prior arts seldom use the generative models, which is usually more promising for expressing structural relationship among random variables. And they may fuse 1-2 data sources in semi-supervised setting , but not scale to even more data sources. In this paper, we resort to a generative model to mimic the messages in a terminals conversation and enrich the target data domain from two abundant but different information sources: labeled but from simulation, and genuine but unlabeled. The transfer and semi-supervised demands are integrated into an intuitive framework, composed of a connected graph of multiple simple Generative Adversarial Networks (GANs)' components, trained in an alternative optimization approach. The contribution of this paper includes: 1) combine three kinds of data sources in a generative approach, to solve the small-sample problem with a simulation environment; 2) extend the two players in usual GANs to a system of multiple ones, still keeping its merit of end-to-end training; 3) verify its effect on our practice problem of packet sequence classification. The left of paper is organized as below: first, we introduce the previous work selectively in network anomaly detection and the research frontier in the generative neural network. Next, we present the model and algorithm in detail with feature design at different levels. The of experiments are followed in Section 4. Finally, we conclude the whole article. The anomaly detection in communication packets has been long-term studied, either for the Quality of Service (QoS) management or instruction detection. The article BID1 summarized the past works based on their applied technologies, which almost cover all popular machine learning methods before 2012.1 The works after that mainly switched to deep learning as it goes popular, which are surveyed by ourselves. In general, all of these keep developing with both two aspects: the level of automation and the ability of models. BID4 started to train neural networks (NN) on labeled data to build models for more categories of anomalies than hard-coded rules, with an expert feature library and three-layer perceptron classifiers. Later, BID0 verified the feasibility of Self Organizing Map (SOM) in an unsupervised scenario, where the unexpected packets were automatically saved for analyzers. BID2 used the online learning to quickly adapt to the newly occurred attacking samples feedbacked by users, without the effort of retraining. BID11 used the self-taught learning to enrich the dataset from unlabeled live stream, and build up a comprehensive feature space for embedding. The enhancement can be observed even using a simple K-Nearest-Neighbors. On the other hand, the neural network models also advance consistently. The early attempts include PCA NNLiu et al. FORMULA3, Wavelet NNSun et al., etc. Yin et al. (2017 used Recurrent Neural Network (RNN) on classifying 5 categories of packet flows, and achieved obviously better than models ignoring the temporal orders. BID16 designed an NN with 6-dimensional manual feature vectors and 3 hidden layers for inherent mapping as input and claimed accuracy improvements after testing. also used self-taught learning, similar to BID11, but with more sophisticated models. It extracted features automatically from unlabeled data by a sparse auto-encoder, and classify them by a single hidden layer perceptron. To our best knowledge, it is the first time we employ the generative neural networks for the semisupervised and transfers learning simultaneously on this problem. The classical GANs compose of two neural components contesting with each other in a zero-sum game BID7 ). It can be extended to more components for the following purposes (but not limited to): 1) reflecting the relationship between multiple random variables. BID12 solved the multi-class classification by adding an extra classifier to denote the conditional probability p(y|X). The newly classifier can shift the burden of predicting labels from identifying fake samples, which is blended in the previous work BID14. This triple-player formulation makes the model even clearer and brings more stableness during training. In multi-view learning, BID3 defined one discriminator for each view and enabled the distribution estimation over all possible output y if any subset of view on a particular input X is fixed. 2) Enhancing the training process. BID8 addressed the mode collapse problem in GANs by training many generators, which envisions to discriminate itself to an additional classifier and fool the original discriminator meanwhile. It improved the steadiness of training significantly. Instead, BID5 used multiple discriminators to ensemble a more powerful one; on the other hand, the training process is retarded by a predefined function (similar to soft-max) on the top of them to match the generator's capability better. In this paper, we only focus on the former purpose. The packet stream in reality is a sequence of attributed events e = {(t i, {a i,j})|i ∈ N, j ∈ N }, where t i is the timestamp sealed by the receiver, and a i,j is a tuple of key-value parsed from packet headers. The label c of e can be K classes of anomalies containing 1 special class for normality. To focus the main aspect of our problem, we prefer to simplify it by two assumptions: 1) anomalies can only happen at one side of the communication, such as server side, to prevent the number of possible situations from blowing up to K 2. It seldom happens that, both terminals have problems simultaneously and produce a complicated interaction that can fully not be diagnosed by a onesided model. In fact, we can train two individual models for both sides, and thus the records from client side can be removed from the train set in experiments.2) The continuous valued (or finegrained in 10 −6 s) timestamps are ignored, while only their ascending index is kept, from 1 to T. We insert dummy packets, which replicate sequence id from the previous one and fill all other fields with 0, to denote an occurence of timeout between two consecutive items. The overall number of dummy packets is informative to models since it indicates how many periods the opposite side has not responded. It is justified because most protocols are indifferent to the exact time intervals during sending/receiving unless they exceed the predefined timeout threshold. The available content of attributes depends on how much effort we want to pay for an accurate inspection of packet headers. The clearer we want to know about the states of every system running, the much we should know about the details of a given protocol. There are 3 levels of feature engineering: 1) raw bytes of headers, which needs little effort; 2) the numerical values (sequence index, integer or Boolean) parsed by a data scheme indicating their positions and necessity; 3) the latent states defined in protocols, based on complete domain knowledge and Finite-State Machine (FSM) -driven analysis. For instance, a packet at level 1 may be only a binary sequence, like 1001... 1000, which is unreadable for humans. At level 2 FIG0, it turns to be a data structure with proper data types, but without semantics defined by the protocol draft. The array of structures can further be arranged into a multi-dimensional vector. At level 3 FIG0, the inherent states are explicitly parsed out, and the sequence finally achieves its most compact representation -a categorical vector. NN can digest all levels above, though an extra effort is needed for discrete values at level 3, discussed later in Sec. 3.2.2. A simplified version of FSM for a sender during TCP's transmission phase. In the typical situation, the sender shuttles between the Ready and Listen state. When there is some error occurred in the direction that data goes, the system will retreat to a previous position and resend the datum; when the acknowledge packet loses in the coming direction, a timeout will be triggered, and the system will resend the last piece of data. In practice, these states are implicitly encoded in the headers, and the computer program has to analyze them based on TCP protocol. Assume that the N -dimensional sequence Y = {y t : t ∈ T} is generated from repetitive applying of a function G on a latent M -dimensional vector z: DISPLAYFORM0 Where c is a categorical variable to switch between the types of anomalies, and θ is the parameters of G. We assume that, if z ∼ N (0, σ 2) and c ∼ Cat(π), the ed Y will conform to a distribution where our observationsŶ come from. In the adversary approach, we guide the training of G by minimizing the distance (cross entropy, etc.) between the p(Y, c) and observedp(Y, c), via the transformation of an extra binary discriminator D r: DISPLAYFORM1 where H is the cross entropy between two distributions. Similarly, we define a function G s that transforms Y to its correspondence Y s in the simulation world, and a function D s to discriminate them from real simulation data: DISPLAYFORM2. For the unlabeled data, we define a D c to distinguish them from the generated samples without labels c: DISPLAYFORM3 Figure 2: The layout of building blocks in our solution. The arrows denote for the information flow among blocks, and they are connected as three forward paths in (a), representing real, simulated and unlabeled information sources colored as orange, green, and blue. The noises driven the whole process is assumed to be sampled from two independent sources, z ∼ N (0, σ 2) and c ∼ Cat(π). The π are estimated directly based on real labeled data. In (b), the white block C is trained with a fixed G, colored with shading.. The overall optimization goal is a convex combination of Eq. 3, 5 and 7: DISPLAYFORM4 where λ s, λ c is the coefficients to adjust the importance of targets. Once we obtain the p(Y, c) in the implicit form of G, the classifier p(c|Y) can be derived with the marginal distribution p(c) according to the Bayes rule. The neural components and their connections are shown in Fig. 2. There are 3 information sinks in Fig. 2a), each of which connects to exact 2 data sources and corresponds to one part of loss function in Eq. 8; minimizing them concurrently, is equivalent to make the data come from different paths look the same with the best effort. Note that the graph G is connected, directed and acyclic, which makes any topological order (must exist based on graph theory) of nodes is viable to a gradient descent approach. It provides a convenience for the design of optimization heuristics in Sec. 3.2.3. The trained G will be frozen in a secondary training process in Fig. 2b ), to consistently offer the classifier C its pseudo-samples with labels until convergence. The neural blocks in Fig. 2 can be built from even more fine-grained modules: 1 an input layer concatenating a vector of z (or y) and an optional one-hot vector for c; 2 a Long Short-Term Memory (LSTM) layer mapping a sequence to a vector; 3 a LSTM layer reversely mapping a vector to a sequence; 4 a fully connected layer; 5 a sigmoid (or softmax) layer classifying real/fake samples (or outputting labels). The generators G is built by 1 + 3, and G s is 2 + 3, and C is 2 + 5; all D, D s, D u share a same structure, which is 2 + 1 + 5. The modules 2, 3 and 4 are kept as one single hidden layer here, after the multi-layer structures have been tried in practice and their improvement was found to be negligible. For the discrete state sequence at feature level 3, it is feasible to map its discrete values into continuous vector globally by Word2vec during preprocessing, since the C is what we finally interested for diagnosis and the intermediate are not visible to end users indeed. We need a heuristics to guide the whole optimization process of G, depicted in Algo. 1. During every mini-batch iterations of training, every time we select a forward path whose loss function contributes most in the Eq. 8, weighted by λs, and update the θ of blocks on the way in the form of gradient descent. The all three sub-loss functions will be updated after G has been modified. Once the process has found a selected target that does not contribute to the overall goal, the update will be rolled back, and the algorithm will switch to others. The failure record for any target will not bring into next batch. For individual blocks, RMSprop is preferred for components containing a recurrent layer, including G, G s and C, while all discriminators use Adam instead. The real labeled data are collected from the archive of fault cases, probed at the core section of a wireless telecom site in service. The problematic TCP streams have 4 categories: 1) uplink packet loss at random, 2) downlink packet loss at random, 3) packet loss periodically when load exceeds capacity, 4) checksum error at random, 5) healthy but false alarmed. The faults in the connection establishment and release phase are not considered here. In the historical records, all we have are 18 samples, unevenly distributed from category 1-5, which are 5, 7, 3, 1, and 2.The unlabeled data are captured from the same site, having 2000 real samples sufficient for training propose. Though its size is much larger than the labeled ones, it can hardly contain valid anomalies by an arbitrary, short-term collection. It provides a reference for the average condition of service quality for a specific site. The simulation is conducted from a mirror environment in labs, with a similar or maybe simplified configuration. The 5 types of anomalies are generated in equal ratio, and each of them has 400 records. The phenomenon of occurring errors here is much more evident than reality: 1) for uplink and 2) downlink packets, the portability of loss is 50%, 3) the stream have 50% of time transmitting over the limit of throughput; 4) the probability of checksum error is 50%.The sequence lengths of all synthetic data are all fixed to 500.The data preprocessing on the TCP header is based on different levels discussed in Sec. 3.1, where the latent states of TCP include normal, resend, and timeout, distilled from 7 useful attributes, and also from 24 bytes of raw binary records. A simple FSM is defined to compress the attributes to several states, according to TCP's standard logic. All sequences are split into the size of 500, and the ones shorter than that are padded by 0. The performance of our multi-class problem is evaluated by the accuracy Acc = N correct /N, which is only metrics cared by operation engineers. They rank all the possible causes descendingly based on the model's output and take the whole list as a recommendation, which acts as a shortcut leading them to the underlying causes much more quickly, especially compared to their existing manualbased approach. We measure two variations of accuracy: averaging on samples as above, and averaging on class, i.e., K c=1 N Nc Acc c, to emphasize the performance on minor-classes. 3-fold cross-validation is used for the set of real labeled data, with all other 2 datasets always assisting the train partition for every fold. The program is based on Keras 2 and a GAN library 3. The one-to-many recurrent layer of Keras is implemented merely in the form of many-to-many, by repeating the input as a vector of equal length as output. The dimension of noise X is set to 20, and hidden dimension of LSTM is 10 with L2 regularization. The learning rates of all components are configured to 10 −3. All other parameters are kept by default. We have two kinds of factors to combine into a workable model: feature levels and data sources, shown in Tab. 1. The solution in Fig. 2a) can be trimmed based on available data sources. In the group 1 of Tab.1, we give two referential models for comparison, one (Line 1) is the dummy model which always give the most frequent class as prediction, and the other (Line 2) is the if we only use the simulation data both for train and test with one standalone classifier. With the deliberately amplified anomalies and evenly distributed classes, the performance can be quite ideal, and to some extent be an empirical upper limit of diagnosis. It can be observed in group 4∼5 that, the simulation data are crucial for substantial enhancement by adding more typical anomalies, while the unlabeled contributes slightly via only supplying normal samples. The improvements in weighted accuracy are more obvious, which is more than twice that of dummy model. On the other hand, the features still act an essential role in our problem. The level 3 feature can always perform better than the rest two levels, while the level 1 can especially not produce any meaningful . The level 2 can approximate the best performance with the help of massive data, offering an always worse but still acceptable accuracy, without the effort of understanding protocols. The evolution of losses of 3 discriminators is demonstrated in FIG2, and the classifier is shown in FIG2 ). All loss curves of GANs' components converge to their roughly horizontal limits, with more or less mutually influenced fluctuations, caused by the continuous attempts to seek for a better equilibrium in every batch. However, these efforts seem to merely make the overall loss tremble, instead of moving to lower places. The variances of simulated data are obviously much smaller than real data, which may be ascribed to the sufficient size of data and evenly distribution among classes, whereas the real data are of imbalance, and have few samples to validate the improvements during training. We terminated the whole process at iteration 10 4, and use the trained G to gain a corresponding C, which is much easier to train as the loss goes steadily to its convergence level, shown in FIG2 ). In this paper, the widely used semi-supervised and transfer learning requirements have been implemented in an integrated way, via a system of cooperative or adversarial neural blocks. Its effectiveness has been verified in our application of packet flow classification, and it is hopeful to be a widely adopted method in this specific domain. The work also prompts us that, complex machine learning tasks and their compound loss functions can be directly mapped into connected networks, and their optimization process can be designed over an entire graph, rather than each individual's hierarchical layers. In future work, we may study how to apply this approach to even larger scale tasks, and make a theoretical analysis of the existence of equilibrium and why we can always reach it.

Represent the TLDR for finding the corresponding scientific article!

semi-supervised and transfer learning on packet flow classification, via a system of cooperative or adversarial neural blocks

Represent the short summary for finding the corresponding scientific article (long)

Considering neural network optimization process as a model selection problem, we introduce a biological plausible normalization method that extracts statistical regularity under MDL principle to tackle imbalanced and limited data issue.