Question: Represent the scientific article to find a short TLDR

Question: Imitation learning aims to inversely learn a policy from expert demonstrations, which has been extensively studied in the literature for both single-agent setting with Markov decision process (MDP) model, and multi-agent setting with Markov game (MG) model. However, existing approaches for general multi-agent Markov games are not applicable to multi-agent extensive Markov games, where agents make asynchronous decisions following a certain order, rather than simultaneous decisions. We propose a novel framework for asynchronous multi-agent generative adversarial imitation learning (AMAGAIL) under general extensive Markov game settings, and the learned expert policies are proven to guarantee subgame perfect equilibrium (SPE), a more general and stronger equilibrium than Nash equilibrium (NE). The experiment demonstrate that compared to state-of-the-art baselines, our AMAGAIL model can better infer the policy of each expert agent using their demonstration data collected from asynchronous decision-making scenarios (i.e., extensive Markov games). Imitation learning (IL) also known as learning from demonstrations allows agents to imitate expert demonstrations to make optimal decisions without direct interactions with the environment. Especially, inverse reinforcement learning (IRL) recovers a reward function of an expert from collected demonstrations, where it assumes that the demonstrator follows an (near-)optimal policy that maximizes the underlying reward. However, IRL is an ill-posed problem, because a number of reward functions match the demonstrated data (Ziebart et al. (2008; ; ;), where various principles, including maximum entropy, maximum causal entropy, and relative entropy principles, are employed to solve this ambiguity (Ziebart et al. (2008; ; ; ;). Going beyond imitation learning with single agents discussed above, recent works including , , have investigated a more general and challenging scenario with demonstration data from multiple interacting agents. Such interactions are modeled by extending Markov decision processes on individual agents to multi-agent Markov games (MGs) (Littman & Szepesvári ). However, these works only work for synchronous MGs, with all agents making simultaneous decisions in each turn, and do not work for general MGs, allowing agents to make asynchronous decisions in different turns, which is common in many real world scenarios. For example, in multiplayer games , such as Go game, and many card games, players take turns to play, thus influence each other's decision. The order in which agents make decisions has a significant impact on the game equilibrium. In this paper, we propose a novel framework, asynchronous multi-agent generative adversarial imitation learning (AMAGAIL): A group of experts provide demonstration data when playing a Markov game (MG) with an asynchronous decision-making process, and AMAGAIL inversely learns each expert's decision-making policy. We introduce a player function governed by the environment to capture the participation order and dependency of agents when making decisions. The participation order could be deterministic (i.e., agents take turns to act) or stochastic (i.e., agents need to take actions by chance). A player function of an agent is a probability function: given the perfectly known agent participation history, i.e., at each previous round in the history, we know which agent(s) participated, it provides the probability of the agent participating in the next round. With the general MG model, our framework generalizes MAGAIL from the synchronous Markov games to (asynchronous) Markov games, and the learned expert policies are proven to guarantee subgame perfect equilibrium (SPE) , a stronger equilibrium than the Nash equilibrium (NE) (guaranteed in). The experiment demonstrate that compared to GAIL and MAGAIL , our AMAGAIL model can better infer the policy of each expert agent using their demonstration data collected from asynchronous decision-making scenarios. Markov games (MGs) are the cases of N interacting agents, with each agent making a sequence of decisions with strategies only depending on the current state. A Markov game 1 is denoted as a tuple (N, S, A, Y, ζ, P, η, r, γ) with a set of states S and N sets of actions. At each time step t with a state s t ∈ S, if the indicator variable I i,t = 1, an agent i is allowed to take an action; otherwise, I i,t = 0, the agent i does not take an action. As a , the participation vector I t = [I 1,t, · · ·, I N,t] indicates active vs inactive agents at step t. The set of all possible participation vectors is denoted as I, namely, I t ∈ I. Moreover, h t−1 = [I 0, · · ·, I t−1] represent the participation history from step 0 to t − 1. The player function Y (governed by the environment) describes the probability of an agent i being allowed to make an action at a step t, given the participation history h t−1, namely, Y (i|h t−1). ζ defines the participation probability of an agent at the initial time step Note that, the player function can be naturally extended to a higher-order form when the condition includes both previous participation history and previous state-action history; thus, it can be adapted to non-Markov processes. The initial states are determined by a distribution η: S →. Let φ denotes no participation, determined by player function Y, the transition process to the next state follows a transition function: P: S × A 1 ∪ {φ} × · · · × A N ∪ {φ} → P(S). Agent i obtains a (bounded) reward given by a function r i: S ×A i → R 2. Agent i aims to maximize its own total expected return R i = ∞ t=0 γ t r i,t, where γ ∈ is the discount factor. Actions are chosen through a stationary and stochastic policy π i: S × A i →. In this paper, bold variables without subscript i denote the concatenation of variables for all the agents, e.g., all actions as a, the joint policy defined as π(a|s) = N i=1 π i (a i |s), r as all rewards. Subscript −i denotes all agents except for i, then (a i, a −i) represents the action of all N agents (a 1, · · ·, a N). We use expectation with respect to a policy π to denote an expectation with respect to the trajectories it generates. For, denotes the following sample process as s 0 ∼ η, I 0 ∼ ζ, I t ∼ Y, a ∼ π(·|s t), s t+1 ∼ P (s t+1 |s t, a), for ∀i ∈ [N]. Clearly, when the player function Y (i|h t−1) = 1 for all agents i's at any time step t, a general Markov game boils down to a synchronous Markov game , where all agents take actions at all steps. To distinguish our work from MAGAIL and be consistent with the literature and , we refer the game setting discussed in MAGAIL as synchronous Markov games (SMGs), and that of our work as Markov games (MGs). In synchronous Markov games (SMGs), all agents make simultaneous decisions at any time step t, with the same goal of maximizing its own total expected return. Thus, agents' optimal policies are interrelated and mutually influenced. Nash equilibrium (NE) has been employed as a solution concept to resolve the dependency across agents, where no agents can achieve a higher expected reward by unilaterally changing its own policy . However, in Markov games (MGs) allowing asynchronous decisions, there exist situations where agents encounter states (subgames) ed from other agents' "trembling-hand" actions. Since the NE does not consider the "trembling-hand" ed states and subgames, when trapped in these situations, agents are not able to make optimal decisions based on their polices under NE. To address this problem, Selten firstly proposed subgame perfect equilibrium (SPE) . SPE ensures NE for every possible subgame of the original game. It has been shown that in a finite or infinite extensive-form game with either discrete or continuous time, best-response strategies all converge to SPE, rather than NE (; ;). In synchronous Markov games, MAGAIL was proposed to learn experts' policies constrained by Nash equilibrium. Since there may exist multiple Nash equilibrium solutions, a maximum causal entropy regularizer is employed to resolve the ambiguity. Thus, the optimal policies can be found by solving the following multi-agent reinforcement learning problem. where, and β is a weight to the entropy regularization term. In practice, the reward function is unknown. MAGAIL applies multi-agent IRL (MAIRL) below to recover experts' reward functions, with ψ as a convex regularizer, Moreover, MAGAIL solves MARL • MAIRL ψ (π E) to inversely learn each expert's policy via applying generative adversarial imitation learning (D wi is a discriminator for agent i that classifies the experts' vs policy trajectories. π θ represent the learned experts' parameterized policies, which generate trajectories with maximized the scores from Extending multi-agent imitation learning to general Markov games is challenging, because of the asynchronous decision making and dynamic state (subgame) participating. In this section, we will tackle this problem using subgame perfect equilibrium (SPE) solution concept. In a Markov game (MG), the Nash equilibrium needs to be guaranteed at each state s ∈ S 3, namely, we apply subgame perfect equilibrium (SPE) solution concept instead. Formally, a set of agent is an SPE if at each state s ∈ S (also considered as a root node of a subgame), no agent can achieve a higher reward by unilaterally changing its policy on the root node or any other descendant nodes of the root node, i.e., ∀i Therefore, our constrained optimization problem is (, Theorem 3.7.2) For an agent i with a probability of taking action a at state s t given a history h t−1, its Q-function is where P r(I t |h t−1) = i:Ii,t=1 Y (i|h t−1) j:Ij,t=0 (1 − Y (j|h t−1)) is the probability of participation vector I t given history h t−1. The constraints in eq. guarantee an SPE, i.e.,. Consistent with MAGAIL the objective has a global minimum of zero under SPE, and π forms SPE if and only if f r (π, v) reaches zero while being a feasible solution. We use AMA-RL(r) to denote the set of policies that form a sSPE under reward function r, and can maximize γ-discounted causal entropy of policies: where q i is defined in eq.. Our objective is to define a suitable inverse operator AMAIRL in analogy to MAIRL in eq.. The key idea of MAIRL is to choose a reward that creates a margin between a set of experts and every other set of policies. However, the constraints in SPE optimization eq. can make this challenging. To that end, we derive an equivalent Lagrangian formulation of eq. to defined a margin between the expected rewards of two sets of policies to capture the "difference". The SPE constraints in eq. state that no agent i can obtain a higher expected reward via 1-step temporal (TD) difference learning. We replace 1-step constraints with (t+1)-step constraints with the solution remaining the same as AMARL. The general idea is consistent with MAGAIL . The detailed derivation is in Appx A.1. The updated (t+1)-step constraints arê By implementing the (t+1)-step formulation eq., we aim to construct the Lagrangian dual of the primal in eq.. Since for any policy π, f r (π,v) = 0 givenv i defined as in Theorem 1 in Appx A.1 (proved in Lemma 1 in Appx A.2), we just focus on the constraints in eq. to get the dual problem where T t i is the set of all length-t trajectories of the form {s (j), a i | · |H| Lagrange multipliers, andv i is defined as in Theorem 1 in Appx A.1. Theorem 2 illustrates that a specific λ is able to recover the difference of the sum of expected rewards between not all optimal and all optimal policies. Theorem 2 For any two policies π * and π, let be the probability of generating the sequence τ i using policy π i and π * −i and h t−1, where P r(h t−1) = P r(I 0) t−1 k=1 P r(I k |h k−1) is the probability of history h t−1. Then where the dual function is L (t+1) r (π *, λ * π) and each multiplier can be considered as the probability of generating a trajectory of agent i ∈ N, τ i ∈ T t i, and h t−1 ∈ H. Theorem 2 (proved in Appx A.3) provides a horizon to establish AMAIRL objective function with regularizer ψ. is the discounted causal entropy for policy π i when other agents follow π E−i, and β is a hyper-parameter controlling the strength of the entropy regularization term as in GAIL . We first define the asynchronous occupancy measure in Markov games: with a policy π i ∈ Π, define its asynchronous occupancy measure ρ The occupancy measure can be interpreted as the distribution of state-action pairs that an agent i encounters under the participating and nonparticipating situations. Notably, when ζ(i) = 1, Y (i|h t−1) = 1 for all t ∈ {1, ..., ∞}, h t−1 ∈ H, asynchronous occupancy measure in MG turns to the occupancy measure defined in MAGAIL and GAIL, i.e., ρ p πi = ρ πi. With the additively separable regularization ψ, for each agent i, π Ei is the unique optimal response to other experts π E−i. Therefore we obtain the following theorem (see proof of Theorem 3 in Appendix A.4):, and that AMA-RL(r) has a unique solution 4 for all r ∈ AMA-IRL ψ (π E), then where π i, E −i denotes π i for agent i, and π E−i for other agents. In practice, we are only able to calculate ρ p π E and ρ p π. As following MAGAIL (In this section, we propose practical algorithms for asynchronous multi-agent imitation learning, and introduce three representative scenarios with different player function structures. The selected ψ i in Proposition 1 (in Appx A.5) contributes to the corresponding generative adversarial model where each agent i has a generator π θi and a discriminator, D wi. When the generator is allowed to behave, the produced behavior will receive a score from discriminator. The generator attempts to train the agent to maximize its score and fool the discriminator. We optimize the following Three agents all have stochastic player functions (i.e., yellow boxes), thus, each agent has a certain probability to make an action w.r.t the player function given the participation history h t−1; in this example, only agents #2 and #3 happen to make actions, and agent #1 does not. objective: In practice, the input of AMAGAIL is Z, the demonstration data from N expert agents in the same environment, where the demonstration data Z = {(s t, a)} T t=0 are collected by sampling s 0 ∼ η, The assumptions include knowledge of N, γ, S, A. Transition P, initial state distribution η, agent distribution ζ, player function Y are all considered as black boxes, and no additional expert interactions with environment during training process are allowed. In the RL process of finding each agent's policy π θi, we follow MAGAIL to apply Multi-agent Actor-Critic with Kronecker-factors (MACK) and use the advantage function with the baseline V ν for variance reduction. The summarized algorithm is presented in Algorithm 1 in Appx B. In MGs, the order in which agents make decisions is determined by the player function Y. Below, we discuss three representative structures of player function Y, including synchronous participation, deterministic participation, and stochastic participation. Synchronous participation. When Y (i|h t−1) = 1 holds for all agents i ∈ [N] at every step t (as shown in Figure 1a), agents make simultaneous actions, and a general Markov game boils down to a simple synchronous Markov game. Deterministic participation. When the player function Y (i|h t−1) is deterministic for all agents i ∈ [N], it can only output 1 or 0 at each step t. Many board games, e.g., Go, and Chess, have deterministic player functions, where agents take turns to play. Figure 1b shows an example of deterministic participation structure. Stochastic participation. When the player function is stochastic, namely, Y (i|h t−1) ∈ for some agent i ∈ [N] at certain time step t, the agent i will make an action by chance. As illustrated in Figure 1c, three agents all have stochastic player functions at step t, and agent #1 does not take an action at step t, while agent #2 and #3 happen to take actions. We evaluate AMAGAIL with both stochastic and deterministic player function structures under cooperative and competitive games, respectively. We compared our AMAGAIL with two baselines, including Behavior Cloning (BC) by OpenAI and decentralized Multi-agent generative adversarial imitation learning (MAGAIL) . The are collected by averaging over 5 random seeds (refer to Appx C for implementation details). We use the particle environment as a basic setting, and customize it into four games to allow different asynchronous player function structures. Deterministic Cooperative Navigation: Three agents (agent #1, #2 and #3) need to cooperate to get close to three randomly placed landmarks through physical actions. They get high rewards if they are close to the landmarks and Figure 2: Average true reward from cooperative tasks. Performance of experts and random policies are normalized to one and zero respectively. We use inverse log scale for better comparison. are penalized for any collision with each other. Ideally, each agent should cover a single distinct landmark. In this process, the agents must follow a deterministic participation order to take actions, i.e., in the first round all three agents act, in the second round only agent #1 and #2 act, in the third round only agent #1 acts, and repeat these rounds until the game is completed. Stochastic Cooperative Navigation: This game is the same with deterministic cooperative navigation except that all three agents have a stochastic player function. Each agent has 50% chance to act at each round t. Deterministic Cooperative Reaching: This game has three agents with their goals as cooperatively reaching a single landmark with minimum collision. In this game, agents follow a deterministic player function, same as that in deterministic cooperative navigation game, to make actions. Stochastic Predator-Prey: Three slower cooperating agents (referred to as adversaries) chase a faster agent in an environment of two landmarks; the faster agent acts first, then each adversary with a stochastic player function of 50% chance to act with the same goal of catching the faster agent. The adversaries and the agent need to avoid two randomly placed landmarks. The adversaries collect rewards when touching the agent, where the agent is penalized. Note that, an agent that does not participate in a round of a game does not get a reward. In these four game environments, agents are first trained with Multi-agent ACKTR; ), thus the true reward functions are available, which enable us to evaluate the quality of recovered policies. When generating demonstrations from well-trained expert agents, a "null" (no-participation) as a placeholder action is recorded for each no-participation round in the trajectory. The quality of a recovered policy is evaluated by calculating agents' average true reward of a set of generated trajectories. We compare our AMAGAIL with two baselines -behavior cloning (BC) and decentralized Multi-agent generative adversarial imitation learning (MAGAIL) . Behavior cloning (BC) utilizes the maximum likelihood estimation for each agent independently to approach their policies. Decentralized multi-agent generative adversarial imitation learning (MAGAIL) treats each agent with a unique discriminator working as the agent's reward signal and a unique generator as the agent's policy. It follows the maximum entropy principle to match agents' occupancy measures from recovered policies to demonstration data. We compare AMAGAIL with baselines under three particle environment games, namely, deterministic cooperative navigation, stochastic cooperative navigation, and deterministic cooperative reaching games. Figure 2 show the normalized rewards, when learning policies with BC, MAGAIL and AMAGAIL, respectively. When there is only a small amount of expert demonstrations, the normalized rewards of BC and AMAGAIL increase, especially, when less demonstration data are used, i.e., less than 400 demonstrations. After a sufficient amount of demonstrations are used, i.e., more than 400, AMAGAIL has higher rewards than BC and MAGAIL. This makes sense since at certain time steps there exist non-participating agents (based on the player functions), but BC and MAGAIL models consider the no-participation as an action the agent can choose, where in reality it is governed by the environment. On the other hand, with the introduced player function Y, AMAGAIL characterizes such no participation events correctly, thus more accurately learns the expert policies. The normalized awards of BC are roughly unchanged in Figure 2 (a)&(c), and in Figure 2 (b) after 400 demonstrations, which seems contradictory to that of; , and can be explained as follows. In Figure 2 (b) (stochastic cooperative navigation), the performance of BC is low when using less demonstrations, but increases rapidly as more demonstrations are used, and finally converges to the "best" performance around 0.65 with 300 demonstrations. In Figure 2 (a) (resp. Figure 2(c) ), deterministic cooperative navigation (resp. reaching) is easier to learn compared Table 1: Average agent rewards in stochastic predator-prey. We compare behavior cloning (BC) and multi-agent GAIL (MAGAIL) methods. Best are marked in bold. Note that high vs low rewards are preferred, when running BC for agent vs adversaries, respectively). Task Stochastic Predator-Prey Agent Behavior Cloning MAGAIL AMAGAIL Adversaries BC MAGAIL AMAGAIL Behavior Cloning Rewards −5.0 ± 10.8 −9.0 ± 13.1 −14.0 ± 19.4 −3.6 ± 8.5 −2.1 ± 6.9 with the stochastic cooperative navigation game shown in Figure 2 (b), since there is no randomness in the player function. The performance with only 200 demonstrations is already stabilized at 0.7 (resp. 0.94). In the stochastic cooperative navigation game (Figure 2(b) ), AMAGAIL performs consistently better than MAGAIL and BC. However, in the deterministic cooperative navigation game (Figure 2(b) ), with 200 demonstration, AMAGAIL does not perform as well as MAGAIL. This is due to the game setting, namely, two players actively searching for landmarks are sufficient to gain a high reward in this game. The last agent, player #3, learned to be "lazy", without any motivation to promote the total shared reward among all agents. In this case, it is hard for AMAGAIL to learn a good policy of player #3 with small amount of demonstration data, because player #3's has 2 3 absence rate, given the pre-defined deterministic participation function. Hence, AMAGAIL does not have enough state-action pairs to learn player #3. This gets improved when there are sufficient data, say, more than 400 demonstrations. When we adjust the game setting from 3 landmarks to 1 landmark, i.e., all agents need to act actively to reach the landmark. This is captured in the deterministic cooperative reaching game. In this scenario, an inactive player will lower down the overall reward. As shown in Figure 2 (c), AMAGAIL outperforms BC and MAGAIL consistently, even with a small amount of demonstration data. 5.2 PERFORMANCE WITH MIXED GAME MODE Now, we further evaluate the performance of AMAGAIL under a mixed game mode with both cooperative and adversarial players, i.e., stochastic predator-prey game. Since there are two competing sides in this game, we cannot directly compare each methods' performance via expected reward. Therefore, we use the's evaluation paradigm and compare with baselines by letting (agents trained by) BC play against (adversaries trained by) other methods, and vice versa. From Table 1, AMAGAIL consistently performs better than MAGAIL and BC. Imitation learning (IL) aims to learn a policy from expert demonstrations, which has been extensively studied in the literature for single agent scenarios . Behavioral cloning (BC) uses the observed demonstrations to directly learn a policy . Apprenticeship learning and inverse reinforcement learning (IRL) (( ; ; Ziebart et al. (2008;)) seek for recovering the underlying reward based on expert trajectories in order to further learn a good policy via reinforcement learning. The assumption is that expert trajectories generated by the optimal policy maximize the unknown reward. Generative adversarial imitation learning (GAIL) and conditional GAIL (cGAIL) incorporate maximum casual entropy IRL and the generative adversarial networks to simultaneously learn non-linear policy and reward functions (; ;). A few recent studies on multi-agent imitation learning, such as MAGAIL (and MAAIRL , model the interactions among agents as synchronous Markov games, where all agents make simultaneous actions at each step t. These works fail to characterize a more general and practical interaction scenario, i.e., Markov games including turn-based games , where agents make asynchronous decisions over steps. In this paper, we make the first attempt to propose an asynchronous multi-agent generative adversarial imitation learning (AMAGAIL) framework, which models the asynchronous decision-making process as a Markov game and develops a player function to capture the participation dynamics of agents. Experimental demonstrate that our proposed AMAGAIL can accurately learn the experts' policies from their asynchronous trajectory data, comparing to state-of-the-art baselines. Beyond capturing the dynamics of participation vs no-participation (as only two participation choices), our proposed player function Y (and AMAGAIL framework) can also capture a more general case 5, where Y determines how the agent participates in a particular round, i.e., which action set A A.1 TIME DIFFERENCE LEARNING Theorem 1. For a certain policy π and reward r, letv i (s (t); π, r, h t−1 ) be the unique solution to the Bellman equation: i; π, r, h t−1 ) as the discounted expected return for the i-th agent conditioned on visiting the trajectory {s (j), a (j) } t−1 j=0, s (t) in the first t − 1 steps and choosing action a (t) i at the t-th step, when other agents using policy π −i: i; π, r, h t−1 ) Then π is subgame perfect equilibrium if and only if: Theorem 1 illustrates that if we replace the 1-step constraints with (t + 1)-step constraints, we still get the same solution as AMA-RL(r) in terms of a subgame perfect equilibrium solution. A.2 EXISTENCE AND EQUIVALENCE OF V AND SUBGAME PERFECT EQUILIBRIUM Lemma 1 By definition ofv i (s (t); π, r, h t−1 ) in Theorem 1 andq i (s (t), a i; π, r, h t−1 ) in eq. 7. Then for any π, f r (π,v) = 0. Furthermore, π is subgame perfect equilibrium under r if and only if v i (s; π, r, h t−1) ≥q i (s, a i ; π, r, h t−1) for all i ∈ [N], s ∈ S, a i ∈ A i and h t−1 ∈ H. Proof We havê i; π, r, h t−1 )]. which utilizes the fact that a i and a −i are independent at s. Therefore, we can easily get f r (π,v) = 0. If π is a subgame perfect equilibrium, and existing one or more of the constrains does not hold, so agent i can receive a strictly higher expected reward for rest of the states, which is against the subgame perfect equilibrium assumption. If the constraints hold, i.e., for all i and (s, a i),v i (s; π, r, h t−1) ≥q i (s, a i ; π, r, h t−1) then Value iteration, thus, overv i (s; π, r, h t−1) converges. If one can find another policy π so that v i (s; π, r, h t−1) < E πi [q i (s, a i ; π, r, h t−1)], then at least one violation exists in the constraints since π i is a convex combination over action a i. Therefore, for any policy π i and action a i for any agent i, E πi [q i (s, a i ; π, r, h t−1)] ≥ E π i [q i (s, a i ; π, r, h t−1)] always hold, so π i is the optimal reply to π −i, and π constitutes a subgame perfect equilibrium once it repeats this argument for all agents. Notably, by assuming f r (π, v) = 0 for some v; if v satisfies the assumptions, then v =v. Proof We use Q *,q *,v * to denote the Q,q andv quantities defined for policy π *. For the two For agent i, τ i and h t−1 we have, λ * π (τ i ; h t−1) · Q * i (τ i ; π *, r, h t−1) = P r(τ i ; h t−1) · Q * i (τ i ; π *, r, h t−1). Therefore, the proof of AMA-RL•AMA-IRL can be derived in a similar fashion with GAIL and MAGAIL (Theorem 3 and Proposition 1 discuss the differences from the single agent scenario similar in . On the one hand, in Theorem 3 we make the assumption that AMA-RL(r) has a unique solution, which is always true in the single agent case due to convexity of the space of the optimal policies. On the other hand, in Proposition 1 we remove the entropy regularizer because here the causal entropy for π i may depend on the policies of the other agents, so the entropy regularizer on two sides are not the same quantity. Specifically, the entropy for the left hand side conditions on π E−i and the entropy for the right hand side conditions on π −i (which would disappear in the single-agent case). B APPENDIX B. ALGORITHM Generate state-action pairs of batch size B from π u through the process: s 0 ∼ η, I 0 ∼ ζ, I t ∼ Y, a ∼ π u (·|s t), s t+1 ∼ P (s t+1 |s t, a); φ is recorded as a placeholder action when an agent does not participate in a round; denote the generated state-action pair set as X. Sample state-action pairs from Z with batch size B; denote the demonstrated state-action pair set as X E. for each agent i = 1, · · ·, N do Filter out state-action pairs (s, φ) from X and X E. Filter out state-action pairs (s, φ) from X and X E. Update ν i to decrease the objective: E X,Y [(V νi (s) − V * (s)) 2 ]. Update θ i by policy gradient with the setting step sizes: E X,Y [θi π θi (a i |s i)A i (s, a)]. C APPENDIX C. EXPERIMENT DETAILS C.1 HYPERPARAMETERS For the particle environment, we follow the setting of MAGAIL to use two layer multiple layer perceptrons with 128 cells in each layer for the policy generator network, value

Represent the short summary for finding the corresponding scientific article (long).

This paper extends the multi-agent generative adversarial imitation learning to extensive-form Markov games.

Represent the TLDR for finding the corresponding scientific article

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model’s prediction. Self-training has mostly been well-studied to classification problems. However, in complex sequence generation tasks such as machine translation, it is still not clear how self-training woks due to the compositionality of the target space. In this work, we first show that it is not only possible but recommended to apply self-training in sequence generation. Through careful examination of the performance gains, we find that the noise added on the hidden states (e.g. dropout) is critical to the success of self-training, as this acts like a regularizer which forces the model to yield similar predictions for similar inputs from unlabeled data. To further encourage this mechanism, we propose to inject noise to the input space, ing in a “noisy” version of self-training. Empirical study on standard benchmarks across machine translation and text summarization tasks under different resource settings shows that noisy self-training is able to effectively utilize unlabeled data and improve the baseline performance by large margin. Deep neural networks often require large amounts of labeled data to achieve good performance. However, acquiring labels is a costly process, which motivates research on methods that can effectively utilize unlabeled data to improve performance. Towards this goal, semi-supervised learning methods that take advantage of both labeled and unlabeled data are a natural starting point. In the context of sequence generation problems, semi-supervised approaches have been shown to work well in some cases. For example, back-translation makes use of the monolingual data on the target side to improve machine translation systems, latent variable models are employed to incorporate unlabeled source data to facilitate sentence compression or code generation . In this work, we revisit a much older and simpler semi-supervised method, self-training , where a base model trained with labeled data acts as a "teacher" to label the unannotated data, which is then used to augment the original small training set. Then, a "student" model is trained with this new training set to yield the final model. Originally designed for classification problems, common wisdom suggests that this method may be effective only when a good fraction of the predictions on unlabeled samples are correct, otherwise mistakes are going to be reinforced . In the field of natural language processing, some early work have successfully applied self-training to word sense disambiguation and parsing (; ;). However, self-training has not been studied extensively when the target output is natural language. This is partially because in language generation applications (e.g. machine translation) hypotheses are often very far from the ground-truth target, especially in low-resource settings. It is natural to ask whether self-training can be useful at all in this case. and Apply f θ to the unlabeled instances U Select a subset S ⊂ {(x, f θ (x))|x ∈ U } Train a new model f θ on S ∪ L 6: until convergence or maximum iterations are reached gains were reported and, to the best of our knowledge, it is still unclear what makes self-training work. did not update the decoder parameters when using pseudo parallel data noting that "synthetic target parts may negatively influence the decoder model of NMT". In this paper, we aim to answer two questions: How does self-training perform in sequence generation tasks like machine translation and text summarization? Are "bad" pseudo targets indeed catastrophic for self-training? If self-training helps improving the baseline, what contributes to its success? What are the important ingredients to make it work? Towards this end, we first evaluate self-training on a small-scale machine translation task and empirically observe significant performance gains over the supervised baseline (§3.2), then we perform a comprehensive ablation analysis to understand the key factors that contribute to its success (§3.3). We find that the decoding method to generate pseudo targets accounts for part of the improvement, but more importantly, the perturbation of hidden states -dropout -turns out to be a crucial ingredient to prevent self-training from falling into the same local optimum as the base model, and this is responsible for most of the gains. To understand the role of such noise in self-training, we use a toy experiment to analyze how noise effectively propagates labels to nearby inputs, sometimes helping correct incorrect predictions (§4.1). Motivated by this analysis, we propose to inject additional noise by perturbing also the input. Comprehensive experiments on machine translation and text summarization tasks demonstrate the effectiveness of noisy self-training. Formally, in conditional sequence generation tasks like machine translation, we have a parallel dataset L = {x i, y i} l i=1 and a large unlabeled dataset U = {x j} l+u j=l+1, where |U | > |L| in most cases. As shown in Algorithm 1, classic self-training starts from a base model trained with parallel data L, and iteratively applies the current model to obtain predictions on unlabeled instances U, then it incorporates a subset of the pseudo parallel data S to update the current model. There are two key factors: Selection of the subset S. S is usually selected based on some confidence scores (e.g. log probability) but it is also possible for S to be the whole pseudo parallel data . Combination of real and pseudo parallel data. A new model is often trained on the two datasets jointly as in back-translation, but this introduces an additional hyper-parameter to weigh the importance of the parallel data relative to the pseudo data . Another way is to treat them separately -first we train the model on pseudo parallel data S, and then fine-tune it on real data L. In our preliminary experiments, we find that the separate training strategy with the whole pseudo parallel dataset (i.e. S = {(x, f θ (x))|x ∈ U }) produces better or equal performance for neural sequence generation while being simpler. Therefore, in the remainder of this paper we use this simpler setting. We include quantitative comparison regarding joint training, separate training, and pseudo-parallel data filtering in Appendix B, where separate training is able to match (or surpass) the performance of joint training. In self-training, the unsupervised loss L U from unlabeled instances is defined as: where p(x) is the empirical data distribution approximated with samples from S, p θ (y|x) is the conditional distribution defined by the model. θ * is the parameter from the last iteration (initially it is set as the parameter of the supervised baseline), and fixed within the current iteration. Eq. 1 reveals the connection between self-training and entropy regularization . 16.8 17.9 ST (baseline) 16.5 17.5 Table 1: Test tokenized BLEU on WMT100K. Self-training are from the first iteration. "Scratch" denotes that the system is initialized randomly and trained from scratch, while "baseline" means it is initialized with the baseline model. In the context of classification, self-training can be understood from the view of entropy regularization , which favors a low-density separation between classes, a commonly assumed prior for semi-supervised learning . To examine the effectiveness of self-training on neural sequence generation, we start by analyzing a machine translation task. We then perform ablation analysis to understand the contributing factors of the performance gains. We work with the standard WMT 2014 English-German dataset consisting of about 3.9 million training sentence pairs after filtering long and imbalanced pairs. Sentences are encoded using 40K byte-pair codes . As a preliminary experiment, we randomly sample 100K sentences from the training set to train the model and use the remaining English sentences as the unlabeled monolingual data. For convenience, we refer to this dataset as WMT100K. Such synthetic setting allows us to have high-quality unlabeled data to verify the performance of self-training. We train with the Base Transformer architecture and dropout rate at 0.3. Full training and optimization parameters can be found in Appendix A.1. All experiments throughout this paper including the transformer implementation are based on the fairseq toolkit, and all are in terms of case-sensitive tokenized BLEU . We use beam search decoding (beam size 5) to create the pseudo targets and to report BLEU on test set. In Figure 1, we use green bars to show the of applying self-training for three iterations. We include both pseudo-training (PT): the first step of self-training where we train a new model (from scratch) using only the pseudo parallel data generated by the current model, and finetuning (FT): the fine-tuned system using real parallel data based on the pretrained model from the PT step. Note that in the fine-tuning step the system is re-initialized from scratch. Surprisingly, we find that the pseudo-training step at the first iteration is able to improve BLEU even if the model is only trained on its own predictions, and fine-tuning further boosts the performance. The test BLEU keeps improving over the first three iterations, until convergence to outperform the initial baseline by 3 BLEU points. This behaviour is unexpected because no new information seems to be injected during this iterative process -target sentences of the monolingual data are from the base model's predictions, thus translation errors are likely to remain, if not magnified. This is different from back-translation where new knowledge may originate from an additional backward translation model and real monolingual targets may help the decoder generate more fluent sentences. One straightforward hypothesis is that the added pseudo-parallel data might implicitly change the training trajectory towards a (somehow) better local optimum, given that we train a new model from scratch at each iteration. To rule out this hypothesis, we perform an ablation experiment and initialize θ from the last iteration (i.e. θ *). Formally, based on Eq. 1 we have: because the conditional log likelihood is maximized when p θ (y|x) matches the underlying data distribution p θ * (y|x). Therefore, the parameter θ should not (at least not significantly) change if we initialize it with θ * from the last iteration. Table 1 shows the comparison of these two initialization schemes at the first iteration. Surprisingly, continuing training from the baseline model also yields an improvement of 1.9 BLEU points, comparable to initializing from random. While stochastic optimization introduces randomness in the training process, it is startling that continuing training gives such a non-trivial improvement. Next, we investigate the underlying reasons for this. To understand why continuing training contradicts Eq. 2 and improves translation performance, we examine possible discrepancies between our assumptions and the actual implementation, and formulate two new hypotheses: H1. Decoding Strategy. According to this hypothesis, the gains come from the use of beam search for decoding unlabeled data. Since our focus is a sequence generation task, we decode y with beam search to approximate the expectation in E y∼p θ * (y|x) log p θ (y|x), yielding a biased estimate, while sampling decoding would in an unbiased Monte Carlo estimator. The in Table 2 demonstrate that the performance drops by 0.5 BLEU when we change the decoding strategy to sampling, which implies that beam search does contribute a bit to the performance gains. This phenomenon makes sense intuitively since beam search tends to generate higher-quality pseudo targets than sampling, and the subsequent cross-entropy training might benefit from implicitly learning the decoding process. However, the decoding strategy hypothesis does not fully explain it, as we still observe a gain of 1.4 BLEU points over the baseline from sampling decoding with dropout. H2. Dropout . Eq. 1 and Eq. 2 implicitly ignore a (seemingly) small difference between the model used to produce the pseudo targets and the model used for training: at test/decoding time the model does not use dropout while at training time dropout noise is injected in the model hidden states. At training time, the model is forced to produce the same (pseudo) targets given the same set of inputs and the same parameter set but various noisy versions of the hidden states. The conjecture is that the additional expectation over dropout noise renders Eq. 2 false. To verify this, we remove dropout in the pseudo training step 2. The in Table 2 indicate that without dropout the performance of beam search decoding drops by 1.2 BLEU, just 0.7 BLEU higher than the baseline. Moreover, the pseudo-training performance of sampling without dropout is almost the same as the baseline, which finally agrees with our intuitions from Eq. 2. In summary, Table 2 suggests that beam-search decoding contributes only partially to the performance gains, while the implicit perturbation -dropout -accounts for most of it. However, it is still mysterious why such perturbation in such large performance gains. If dropout is meant to avoid overfitting and fit the target distribution better in the pseudo-training step, why does it bring advantages over the baseline given that the target distribution is from the baseline model itself? This is the subject of the investigation in the next section. One hypothesis as to why noise (perturbation) is beneficial for self-training, is that it enforces local smoothness for this task, that is, semantically similar inputs are mapped to the same or similar targets. Since the assumption that similar input should ideally produce similar target largely holds for most tasks in practice, this smoothing effect of pseudo-training step may provide a favorable regularization for the subsequent finetuning step. Unlike standard regularization in supervised training which is local to the real parallel data, self-training smooths the data space covered by the additional and much larger monolingual data. To verify this hypothesis more easily, we work with the toy task of summing two integers in the range 0 to 99. We concatenate the two integers and view them as a sequence of digits, the sum is also predicted at the digit level, thus this is still a sequence to sequence task. There are 10000 possible data points in the entire space, and we randomly sample 250 instances for training, 3 100 for validation, 5000 for test, and 4000 as the unlabeled data. Test errors are computed as the absolute difference between the predicted integer and the ground-truth integer. We use an LSTM model to tackle this task. We perform self-training for one iteration on this toy sum dataset and initialize the model with the base model to rule out differences due to the initialization. Setup details are in Appendix A.1. For any integer pair (x 1, x 2), we measure local smoothness as the standard deviation of the predictions in a 3 × 3 neighborhood of (x 1, x 2). These values are averaged over all the 10000 points to obtain the overall smoothness. We compare smoothness between baseline and ST pseudo-training in Table 3. To demonstrate the effect of smoothing on the fine-tuning step, we also report test errors after fine-tuning. We observe that ST pseudo-training attains better smoothness, which helps reducing test errors in the subsequent fine-tuning step. One natural question is whether we could further improve performance by encouraging even lower smoothness value, although there is a clear trade-off, as a totally smooth model that outputs a constant value is also a bad predictor. One way to decrease smoothness is by increasing the dropout probability in the pseudo-training step, but a large dropout (like 0.5) makes the model too unstable and slow at converging. Therefore, we consider a simple model-agnostic perturbation processperturbing the input, which we refer to as noisy self-training (noisy ST). If we perturb the input during the pseudo-training step, then Eq. 1 would be modified to: where g(x) is a perturbation function. Note that we apply both input perturbation and dropout in the pseudo-training step for noisy ST throughout the paper, but include ablation analysis in §4.3. We first validate noisy ST in the toy sum task. We shuffle the two integers in the input as the perturbation function. Such perturbation is suitable for this task since it would help the model learn the commutative law as well. To check that, we also measure the symmetry of the output space. Specifically, for any point (x 1, x 2), we compute |f (x 1, x 2) − f (x 2, x 1)| and average it over all the points. Both smoothness and symmetry values are reported in Table 3. While we do not explicitly perturb the input at nearby integers, the shuffling perturbation greatly improves the smoothness metric as well. Furthermore, predictions are more symmetric and test errors are reduced. In order to illustrate the effect of smoothness, in Figure 2 we show two examples of error heat map. 4 When a point with large error is surrounded by points with small errors, the labels might propagate due to smoothing and its error is likely to become smaller, ing in a "self-correcting" behaviour, as demonstrated in the left example of Figure 2. However, the prediction of some points might become worse due to the opposite phenomenon too, as shown in the right example of Figure 2. Therefore, the smoothing effect by itself does not guarantee a performance gain in the pseudotraining step, but fine-tuning benefits from it and seems to consistently improve the baseline in all datasets we experiment with. Next, we apply noisy self-training to the more realistic WMT100 translation task. We try two different perturbation functions: Synthetic noise as used in unsupervised MT , where the input tokens are randomly dropped, masked, and shuffled. We use the default noising parameters as in unsupervised MT but study the influence of noise level in §5.4. Paraphrase. We translate the source English sentences to German and translate it back to obtain a paraphrase as the perturbation. Figure 1 shows the over three iterations. Noisy ST (NST) greatly outperforms the supervised baseline by over 6 BLEU points and normal ST by 3 BLEU points, while synthetic noise does not exhibit much difference from paraphrasing. Since synthetic noise is much simpler and more general, in the remaining experiments we use synthetic noise unless otherwise specified. Next, we report an ablation analysis of noisy ST when removing dropout at the pseudo-training step in Table 2. Noisy ST without dropout improves the baseline by 2.3 BLEU points and is comparable to normal ST with dropout. When combined together, noisy ST with dropout produces another 1.4 BLEU improvement, indicating that the two perturbations are complementary. Our experiments below are designed to examine whether the noisy self-training is generally useful across different sequence generation tasks and resource settings. To this end, we conduct experiments on two machine translation datasets and one text summarization dataset to test the effectiveness under both high-resource and low-resource settings. Table 4: Results on two machine translation datasets. For WMT100K, we use the remaining 3.8M English and German sentences from training data as unlabeled data for noisy ST and BT, respectively. We run noisy self-training for three iterations or until performance converges. The model is trained from scratch in the pseudo-training step at each iteration since we found this strategy to work slightly better empirically. Full model and training details for all the experiments can be found in Appendix A.1. In some settings, we also include back-translation as a reference point, since this is probably the most successful semi-supervised learning method for machine translation. However, we want to emphasize that BT is not directly comparable to ST since they use different resources (ST utilizes the unlabeled data on the source side while BT leverages target monolingual data) and use cases. For example, BT is not very effective when we translate English to extremely low-resource languages where there is almost no in-domain target monolingual data available. We follow the practice in to implement BT where we use unrestricted sampling to translate the target data back to the source. Then, we train the real and pseudo parallel data jointly and tune the upsampling ratio of real parallel data. We test the proposed noisy self-training on a high-resource translation benchmark: WMT14 EnglishGerman and a low-resource translation benchmark: FloRes English-Nepali. • WMT14 English-German: In addition to WMT100K, we also report with all 3.9M training examples. For WMT100K we use the Base Transformer architecture, and the remaining parallel data as the monolingual data. For the full setting, we use the Big Transformer architecture and randomly sample 20M English sentences from the News Crawl corpus for noisy ST. • FloRes English-Nepali: We evaluate noisy self-training on a low-resource machine translation dataset FloRes (Guzmán et al., 2019) from English (en) to Nepali (ne), where we have 560K training pairs and a very weak supervised system that attains BLEU smaller than 5 points. For this dataset we have 3.6M Nepali monolingual instances in total (for BT) but 68M English Wikipedia sentences. 5 We randomly sample 5M English sentences for noisy ST. We use the same transformer architecture as in (Guzmán et al., 2019). The overall are shown in Table 4. For almost all cases in both datasets, the noisy ST outperforms the baselines by a large margin (1 ∼ 5 BLEU scores), and we see that noisy ST still improves the baseline even when this is very weak. Effect of Domain Mismatch. Test sets of the FloRes benchmark were built with mixed originaltranslationese -some sentences are from English sources and some are from Nepali sources. Intuitively, English monolingual data should be more in-domain with English-origin sentences and Nepali monolingual data should help more for Nepali-origin sentences. To demonstrate this possible domain-mismatch effect, in Table 4 we report BLEU on the two different test sets separately. As expected, ST is very effective when the source sentences originate from English. Comparison to Back-Translation. Table 4 shows that noisy ST is able to beat BT on WMT100K and on the en-origin test set of FloRes. In contrast, BT is more effective on the ne-origin test set according to BLEU, which is not surprising as the ne-origin test is likely to benefit more from Nepali than English monolingual data. We further evaluate noisy self-training on the Gigaword summarization dataset that has 3.8M training sentences. We encode the data with 30K byte-pair codes and use the Base Transformer architecture. Similar to the setting of WMT100K, for Gigaword we create two settings where we sample 100K or 640K training examples and use the remaining as unlabeled data to compare with BT. We also consider the setting where all the 3.8M parallel samples are used and we mine in-domain monolingual data by revisiting the original preprocessing procedure 7 and using the ∼4M samples that disregarded because they had low-quality targets. We report ROUGE scores in Table 5. Noisy ST consistently outperforms the baseline in all settings, sometimes by a large margin (100K and 640K). It outperforms BT with 100K parallel data but underperforms with 640K parallel data. We conjecture that BT is still effective in this case because the task is still somewhat symmetric as Gigaword mostly contains short sentences and their compressed summaries. Notably, noisy ST in the full setting approaches the performance of state-of-the-art systems which use much larger datasets for pretraining . In this section, we focus on the WMT English-German dataset to examine the effect of three factors on noisy self-training: the size of the parallel dataset, the size of the monolingual dataset, and the noise level. All the noisy ST are after the fine-tuning step. Parallel data size. We fix the monolingual data size as 20M from News Crawl dataset, and vary the parallel data size as shown in Figure 3(a). We use a small LSTM model for 10K, Base Transformer for 100K/640K, and Big Transformer for 3.9M. 8 Noisy ST is repeated for three iterations. We see that in all cases noisy ST is able to improve upon the baseline, while the performance gain is larger for intermediate value of the size of the parallel dataset, as expected. Monolingual data size. We fix the parallel data size to 100K samples, and use the rest 3.8M English sentences from the parallel data as monolingual data. We sample from this set 100K, 500K, 1.5M, and 3.8M sentences. We also include another point that uses 20M monolingual sentences from a subset of News Crawl dataset. We report performance at the first iteration of noisy ST. PT FT parallel baseline -15.6 noisy ST, 100K mono + fake target 10.2 16.6 noisy ST, 3.8M mono + fake target 16.6 19.3 noisy ST, 100K parallel + real target 6.7 11.3 noisy ST, 100K parallel + fake target 10.4 16.0 Table 6: Results on WMT100K data. All are from one single iteration. "Parallel + real/fake target" denotes the noise process applied on parallel data but using real/fake target in the "pseudo-training" step. " Mono + fake target" is the normal noisy self-training process described in previous sections. Figure 3(b) illustrates that the performance keeps improving as the monolingual data size increases, albeit with diminishing returns. Noise level. We have shown that noisy ST outperforms ST, but intuitively larger noise must not always be better since at some point it may destroy all the information present in the input. We adopt the WMT100K setting with 100K parallel data and 3.8M monolingual data, and set the word blanking probability in the synthetic noise to 0.2 (default number), 0.4, 0.6, and 0.8. We also include the baseline ST without any synthetic noise. Figure 3 (c) demonstrates that performance is quite sensitive to noise level, and that intermediate values work best. It is still unclear how to select the noise level a priori, besides the usual hyper-parameter search to maximize BLEU on the validation set. In this section, we justify whether the proposed noisy self-training process would help the supervised baseline alone without the help of any monolingual data. Similar to the training process on the monolingual data, we first train the model on the noisy source data (pseudo-training), and then finetune it on clean parallel data. Different from using monolingual data, there are two variations here in the "pseudo-training" step: we can either train with the fake target predicted by the model as on monolingual data, or train with the real target paired with noisy source. We denote them as "parallel + fake target" and "parallel + real target" respectively, and report the performance on WMT100K in Table 6. We use the same synthetic noise as used in previous experiments. When applying the same noise process to parallel data using fake target, the smoothing effect is not significant since it is restricted into the limited parallel data space, producing marginal improvement over the baseline (+0.4 BLEU). As a comparison, 100K monolingual data produces +1.0 BLEU and the effect is enhanced when we increase the monolingual data to 3.8M, which leads to +3.7 BLEU. Interestingly, pairing the noisy source with real target in much worse performance than the baseline, which implies that the use of fake target predicted by the model (i.e. distillation) instead of real target is important for the success of noisy self-training, at least in the case where parallel data size is small. Intuitively, the distilled fake target is simpler and relatively easy for the model to fit, but the real target paired with noisy source makes learning even harder than training with real target and real source, which might lead to a bad starting point for fine-tuning. This issue would be particularly severe when the parallel data size is small, in that case the model would have difficulties to fit real target even with clean source. 6 RELATED WORK Self-training belongs to a broader class of "pseudo-label" semi-supervised learning approaches. These approaches all learn from pseudo labels assigned to unlabelled data, with different methods on how to assign such labels. For instance, co-training learns models on two independent feature sets of the same data, and assigns confident labels to unlabeled data from one of the models. Co-training reduces modeling bias by taking into account confidence scores from two models. In the same spirit, democratic co-training or tri-training trains multiple models with different configurations on the same data feature set, and a subset of the models act as teachers for others. Another line of more recent work perturb the input or feature space of the student's inputs as data augmentation techniques. Self-training with dropout or noisy self-training can be viewed as an instantiation of this. These approaches have been very successful on classification tasks (; ; ; ;) given that a reasonable amount of predictions of unlabeled data (at least the ones with high confidence) are correct, but their effect on language generation tasks is largely unknown and poorly understood because the pseudo language targets are often very different from the ground-truth labels. Recent work on sequence generation employs auxiliary decoders when processing unlabeled data, overall showing rather limited gains. In this paper we revisit self-training for neural sequence generation, and show that it can be an effective method to improve generalization, particularly when labeled data is scarce. Through a comprehensive ablation analysis and synthetic experiments, we identify that noise injected during self-training plays a critical role for its success due to its smoothing effect. To encourage this behaviour, we explicitly perturb the input to obtain a new variant of self-training, dubbed noisy selftraining. Experiments on machine translation and text summarization demonstrate the effectiveness of this approach in both low and high resource settings. For all experiments, we optimize with Adam using β 1 = 0.9, β 2 = 0.98, = 1e − 8. All implementations are based on fairseq, and we basically use the same learning rate schedule and label smoothing as in fairseq examples to train the transformers. 9 Except for the toy sum dataset which we runs on a single GPU and each batch contains 32 examples, all other experiments are run on 8 GPUs with an effective batch size of 33K tokens. All experiments are validated with loss on the validation set. For self-training or noisy self-training, the pseudo-training takes 300K synchronous updates while the fine-tuning step takes 100K steps. We use the downloading and preprocessing scripts in fairseq to obtain the WMT 2014 EnglishGerman dataset, 10 which hold out a small fraction of the original training data as the validation set. The model architecture for the toy sum dataset is a single-layer LSTM with word embedding size 32, hidden state size 32, and dropout rate 0.3. The model architecture of WMT10K baseline in Figure 3 (a) is a single layer LSTM with word embeddings size 256, hidden state size 256, and dropout rate 0.3. We provide more details and evidence to show that our baseline model on WMT100K dataset is trained properly. In all the experiments on WMT100K dataset including baseline and self-training ones, we use Adam optimizer with learning rate 0.0005, which is defaulted in fairseq. We do not use early stop during training but select the best model in terms of the validation loss. We train with 30K update steps for the baseline model and (300K pseudo-training + 100K fine-tuning) update steps for self-training. In both cases we verified that the models are trained sufficiently to fully converge through observing the increase of validation loss. Figure 4 shows the validation curve of the baseline model. Note that the model starts to overfit, and we select the model checkpoint at the lowest point. We also varied the learning rate hyperparameter as 0.0002, 0.0005, and 0.001, which produced BLEU score 15.0, 15.6 (reported in the paper), and 15.5 respectively -our baseline model in previous sections obtained the best performance. In the paper we perform self-training with separate pseudo-training and fine-tuning steps and always use all monolingual data. However, there are other variants such as joint training or iteratively adding confident examples. Here we compare these variants on WMT100K dataset, noisy self-training uses paraphrase as the perturbation function. For joint training, we tune the upsampling ratio of parallel data just as in back-translation . We perform noisy self-training for 3 iterations, and for filtering experiments we iteratively use the most confident 2.5M, 3M, and 3.8M monolingual data respectively in these 3 iterations. Table 7 shows that the filtering process helps joint training but still underperforms separate-training methods by over 1.5 BLEU points. Within separate training filtering produces comparable to using all data. Since separate training with all data is the simplest method and produces the best performance, we stick to this version in the paper. We additionally show the error heat maps of the entire data space on the toy sum datasets for the first two iterations. Here the model at pseudo-training step is initialized as the model from last iteration to clearly examine how the decodings change due to injected noise. As shown in Figure 5, for each iteration the pseudo-training step smooths the space and fine-tuning step benefits from it and greatly reduces the errors 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We revisit self-training as a semi-supervised learning method for neural sequence generation problem, and show that self-training can be quite successful with injected noise.

We compare many tasks and task combinations for pretraining sentence-level BiLSTMs for NLP tasks. Language modeling is the best single pretraining task, but simple baselines also do well.