Question: Represent the scientific article to find a short TLDR

Question: We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of very off-policy learning. We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module. To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods. Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable. We provide extensive empirical validation of the proposed solution. We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames. Value-based and actor-critic policy gradient methods are the two leading techniques of constructing general and scalable reinforcement learning agents . Both have been combined with non-linear function approximation , and have achieved remarkable successes on multiple challenging domains; yet, these algorithms still require large amounts of data to determine good policies for any new environment. To improve data efficiency, experience replay agents store experience in a memory buffer (replay) , and reuse it multiple times to perform reinforcement learning updates . Experience replay allows to generalize prioritized sweeping to the non-tabular setting , and can also be used to simplify exploration by including expert (e.g., human) trajectories . Overall, experience replay can be very effective at reducing the number of interactions with the environment otherwise required by deep reinforcement learning algorithms . Replay is often combined with the value-based Q-learning , as it is an off-policy algorithm by construction, and can perform well even if the sampling distribution from replay is not aligned with the latest agent's policy. Combining experience replay with actor-critic algorithms can be harder due to their on-policy nature. Hence, most established actor-critic algorithms with replay such as (; ;) employ and maintain Q-functions to learn from the replayed off-policy experience. In this paper, we demonstrate that off-policy actor-critic learning with experience replay can be achieved without surrogate Q-function approximators using V-trace by employing the following approaches: a) off-policy replay experience needs to be mixed with a proportion of on-policy experience. We show experimentally (Figure 2) and theoretically that the V-trace policy gradient is otherwise not guaranteed to converge to a locally optimal solution. b) a trust region scheme (; ; can mitigate bias and enable efficient learning in a strongly off-policy regime, where distinct agents share experience through a commonly shared replay module. Sharing experience permits the agents to benefit from parallel exploration (Figures 1 and 3). Our paper is structured as follows: In Section 2 we revisit pure importance sampling for actor-critic agents and V-trace, which is notable for allowing to trade off bias and variance in its estimates. We recall that variance reduction is necessary (Figure 4 left) but is biased in V-trace. We derive proposition 2 stating that off-policy V-trace is not guaranteed to converge to a locally optimal solution -not even in an idealized scenario when provided with the optimal value function. Through theoretical analysis (Section 3) and experimental validation (Figure 2) we determine that mixing on-policy experience into experience replay alleviates the problem. Furthermore we propose a trust region scheme (; ; in Section 4 that enables efficient learning even in a strongly off-policy regime, where distinct agents share the experience replay module and learn from each others experience. We define the trust region in policy space and prove that the ing estimator is correct (i.e. estimates an improved return). As a , we present state-of-the-art data efficiency in Section 5 in terms of median human normalized performance across 57 Atari games , as well as improved learning efficiency on DMLab30 (Table 1). Figure 1: Sharing experience between agents leads to more efficient hyper-parameter sweeps on 57 Atari games. Prior art are presented as horizontal lines (with scores cited from , and). Note that the only previous agent "R2D2" that achieved a score beyond 400% required more than 3,000 million environment steps (see , page 14, Figure 9). We present the pointwise best agent from hyper-parameter sweeps with and without experience replay (shared and not shared). Each sweep contains 9 agents with different learning rate and entropy cost combinations. Replay experiment were repeated twice and ran for 50M steps. To report scores at 200M we ran the baseline and one shared experience replay agent for 200M steps. Table 1: Comparison of state-of-the-art agents on 57 Atari games trained up until 200M environment steps (per game) and DMLab-30 trained until 10B steps (multi-task; all games combined). The first two rows are quoted from and , the third is our implementation of a pixel control agent from and the last two rows are our proposed LASER (LArge Scale Experience Replay) agent. All agents use hyper-parameter sweeps expect for the marked. V-trace importance sampling is a popular off-policy correction for actor-critic agents . In this section we revisit how V-trace controls the (potentially infinite) variance that arises from naive importance sampling. We note that this comes at the cost of a biased estimate (see Proposition 1) and creates a failure mode (see Proposition 2) which makes the policy gradient biased. We discuss our solutions for said issues in Section 4. Figure 2: Left: Learning entirely off-policy from experience replay fails, while combining on-policy data with experience replay leads to improved data efficiency: We present sweeps on DMLab-30 with experience replays of 10M capacity. A ratio of 87.5% implies that there are 7 replayed transitions in the batch for each online transition. Furthermore we consider an agent identical to "LASER 87.5% replay" which however draws all samples from replay. Its batch thus does not contain any online data and we observe a significant performance decrease (see Proposition 2 and 3). The shading represents the point-wise best and worst replica among 3 repetitions. The solid line is the mean. Right: The effect of capacity in experience replay with 87.5% replay data per batch on sweeps on DMLab-30. Data-efficiency improves with larger capacity. Figure 3: Left: Naively sharing experience between distinct agents in a hyper-parameter sweep fails (green) and is worse than the no-replay baseline (blue). The proposed trust region estimator mitigates the issue (red). Right: Combining population based training with trust region estimation improves performance further. All replay experiments use a capacity of 10 million observations and 87.5% replay data per batch. We follow the notation of where an agent interacts with its environment, to collect rewards. On each discrete time-step t, the agent selects an action a t; it receives in return a reward r t and an observation o t+1, encoding a partial view of the environment's state s t+1. In the fully observable case, the RL problem is formalized as a Markov Decision Process : a tuple (S, A, p, γ), where S, A denotes finite sets of states and actions, p models rewards and state transitions (so that r t, s t+1 ∼ p(s t, a t)), and γ is a fixed discount factor. A policy is a mapping π(a|s) from states to action probabilities. The agent seeks an optimal policy π * that maximizes the value, defined as the expectation of the cumulative discounted returns Off-policy learning is the problem of finding, or evaluating, a policy π from data generated by a different policy µ. This arises in several settings. Experience replay mixes data from multiple iterations of policy improvement. In large-scale RL, decoupling acting from learning causes the experience to lag behind the latest agent policy. Finally, it is often useful to learn multiple general value functions (; ; ; ; b) or options from a single stream of experience. On-policy n-step bootstraps give more accurate value estimates in expectation with larger n . They are used in many reinforcement learning agents (; ;). Unfortunately n must be chosen suitably as the estimates variance increases with n too. It is desirable to obtain benefits akin to n-step returns in the off-policy case. To this end multi-step importance sampling can be used. This however adds another source of (potentially infinite ) variance to the estimate. Importance sampling can estimate the expected return V π from trajectories sampled from µ = π, as long as µ is non-zero whereever π is. We employ a previously estimated value function V as a bootstrap to estimate expected returns. , a multi-step formulation of the expected return is where E µ denotes the expectation under policy µ up to an episode termination, δ t V = r t + γV (s t+1) − V (s t) is the temporal difference error in consecutive states s t+1, s t, and π t = π t (a t |s t). Importance sampling estimates can have high variance. Tree Backup , and Q(λ) address this, but reduce the number of steps before bootstrapping even when this is undesirable (as in the on-policy case). RETRACE makes use of full returns in the on-policy case, but it introduces a zero-mean random variable at each step, adding variance to empirical estimates in both on-and off-policy cases. V-trace reduces the variance of importance sampling by trading off variance for a biased estimate of the return -ing in a failure mode (see Proposition 2). It uses clipped importance sampling ratios to approximate i=0 c i ρ t δ t+k V where V is a learned state value estimate used to bootstrap, and ρ t = min [π t /µ t,ρ], c t = min [π t /µ t,c] are the clipped importance ratios. Note that, differently from RETRACE, V-trace fully recovers the Monte Carlo return when on policy. It similarly reweights the policy gradient as: Note that ∇Vπ(s t) recovers the naively importance sampled policy gradient forρ → ∞. In the literature, it is common to subtract a baseline from the action-value estimate r t + γVπ(s t+1) to reduce variance , omitted here for simplicity. The constantsρ ≥c ≥ 1 (typically chosenρ =c = 1) define the level of clipping, and improve stability by ensuring a bounded variance. For any givenρ, the bias introduced by V-trace in the value and policy gradient estimates increases with the difference between π and µ. We analyze this in the following propositions. Proposition 1. The V-trace value estimate Vπ is biased: It does not match the expected return of π but the return of a related implied policyπ defined by equation 3 that depends on the behaviour policy µ:π Proof.. Note that the biased policyπ µ can be very different from π. Hence the V-trace value estimate Vπ may be very different from V π as well. As an illustrative example, consider two policies over a set of two actions, e.g. "left" and "right" represented as a tuple of probabilities. Let us investigate µ = (φ, 1 − φ) and π = (1 − φ, φ) defined for any suitably small φ ≤ 1. Observe that π and µ share no trajectories (state-action sequences) in the limit as φ → 0 and they get more focused on one action. A practical example of this could be two policies, one almost always taking a left turn and one always taking the right. Given sufficient data of either policy it is possible to estimate the value of the other e.g. with naive importance sampling. However observe that V-trace withρ = 1 will always estimate a biased value -even given infinite data. Observe that min [µ(a|x), π(a|x)] = min [φ, 1 − φ] for both actions. Thusπ µ is uniform rather than resembling π the policy. The V-trace estimate Vπ would thus compute the average value of "left" and "right" -poorly representing the true V π. Proposition 2. The V-trace policy gradient is biased: given the the optimal value function V * the V-trace policy gradient does not converge to a locally optimal π * for all off-policy behaviour distributions µ. Proof. See Appendix C. In Proposition 2 we presented a failure mode in V-trace where the variance reduction biases the value estimate and policy gradient. V-trace computes biased Q-estimates Q ω = Q ing in a wrong local policy gradient: The question of how biased the ing policy will be depends on whether the distortion changes the argmax of the Q-function. Little distortions that do not change the argmax will in the same local fixpoint of the policy improvement. The policy will continue to select the optimal action and it will not be biased at this state. The policy will however be biased if the Q-function is distorted too much. For example consider a ω(s, a) that swaps the argmax for the 2nd largest value, the regret will then be the difference between the maximum and the 2nd largest value. Intuitively speaking the more distorted the Q ω, the larger will be the regret compared to the optimal policy. More precisely, the regret of learning a policy that maximizes the distorted Q ω at state s is: where a * = argmax b (Q, b) is the optimal action according to the real Q and, is the optimal action according to the distorted Q ω. For generality, we denote A * as the set of best actions -covering the case with multiple with identical optimal Q-values. Proposition 3 provides a mitigation: Clearly the V-trace policy gradient will converge to the same solution as the true on-policy gradient if the argmax of the Q-function is preserved at all states in a tabular setting. We show that this can be achieved by mixing a sufficient proportion α of on-policy experience into the computation. We show in equation 13 in the Appendix that choosing α such that will in a policy that correctly chooses the best action at state s. Note that Intuitively: the larger the action value gap of the real Q-function Q(s, a *) − Q(s, b) the lower the right hand side and the less on-policy data is required. is negative, then α may be as small as zero and we enabling even pure off-policy learning. Finally note that the right hand side decreases due to d µ (s)/d π (s) if π visits the state s more often than µ. All of those conditions can be computed and checked if an accurate Q-function and state distribution is accessible. How to use imperfect Q-function estimates to adaptively choose such an α remain a question for future research. We provide experimental evidence for these with function approximators in the 3-dimensional simulated environment DMLab-30 with various α ≥ 1/8 in Section 5.3 and Figure 2. We observe that α = 1/8 is sufficient to facilitate stable learning. Furthermore it in better data-efficiency than pure on-policy learning as it utilizes off-policy replay experience. Proposition 3. Mixing on-policy data into the V-trace policy gradient with the ratio α reduces the bias by providing a regularization to the implied state-action values. In the general function approximation case it changes the off-policy V-trace policy gradient from is a regularized stateaction estimate and d π, d µ are the state distributions for π and µ. Note that there exists α ≤ 1 such that Q α has the same argmax (i.e. best action) as Q. Proof. See Appendix C. Mixing online data with replay data has also been argued for by , as a heuristic way of reducing the sensitivity of reinforcement learning algorithms to the size of the replay memory. Proposition 3 grounds this in the theoretical properties of V-trace. To mitigate the bias and variance problem of V-trace and importance sampling we propose a trust region scheme that adaptively selects only suitable behaviour distributions when estimating the state-value of π. To this end we introduce a behaviour relevance function that classifies behaviour as relevant. We then define a trust-region estimator that computes expectations (such as expected returns, or the policy gradient) only on relevant transitions. In proposition 4 and 5 we show that this trust region estimator indeed computes new state-value estimates that improve over the current value function. While our analysis and proof is general we propose a suitable behaviour relevance function in section 4.3 that employs the Kullback Leibler divergence between target policy π and implied policyπ µ: KL (π(·|s)||π µ (·|s)). We provide experimental validation in Figure 3. In off-policy learning we often consider a family of behaviour policies either indexed by training iteration t: M T = {µ t |t < T} for experience replay, or by a different agent k: M K = {µ k |k ∈ K} when training multiple agents. In the classic experience replay case we then sample a time t and locate the transition τ that was generated earlier via µ t. This extends naturally to the multiple agent case where we sample an agent index k and then obtain a transition for such agent or tuples of (k, t). Without loss of generality we simplify this notation and index sampled behaviour policies by a random variable z ∼ Z that represents the selection process. While online reinforcement learning algorithms process transitions τ ∼ π, off-policy algorithms process τ ∼ µ z for z ∼ Z. In this notation, given equation and a bootstrap V, the expectation of importance sampled off-policy returns at state s t is described by: where Above E µz|z represents the expectation of sampling from a given µ z. The conditioning on z is a notational reminder that this expectation does not sample z or µ z but experience from µ z. For any sampled z we obtain a µ z and observe that the inner expectation wrt. experience of µ z in equation recovers the expected on-policy return in expectation: Thus. This holds provided that µ z is non-zero wherever π is. This fairly standard assumption leads us straight to the core of the problem: it may be that some behaviours µ z are ill-suited for estimating the inner expectation. However, standard importance sampling applied to very off-policy experience divides by small µ ing in high or even infinite variance. Similarly, V-trace attempts to compute an estimate of the return following π ing in limited variance at the cost of a biased estimate in turn. The key idea of our proposed solution is to compute the return estimate for π at each state only from a subset of suitable behaviours µ z: M β,π (s) = {µ z |z ∈ Z and β(π, µ, s) < b} as determined by a behaviour relevance function β(π, µ, s): (M Z, M Z, S) → R and a threshold b. The behaviour relevance function decides if experience from a behaviour is suitable to compute an expected return for π. It can be chosen to control properties of V π mix by restricting the expectation on subsets of Z. In particular it can be used to control the variance of an importance sampled estimator: Observe that the inner expectation E µz G π,µ (s t) z in equation already matches the expected return V π. Thus we can condition the expectation on arbitrary subsets of Z without changing the expected value of V π mix. This allows us to reject high variance G π,µ without introducing a bias in V π mix. The same technique can be applied to V-trace where we can reject return estimates with high bias. Using a behaviour relevance function β(s) we can define a trust region estimator for regular importance sampling (IS) and V-trace and show their correctness. We define the trust region estimator as the conditional expectation with λ-returns G, chosen as G IS for importance sampling and G Vtrace for V-trace: where λ π,µ (s t) is designed to constraint Monte-Carlo bootstraps to relevant behaviour: λ π,µ (s t) = 1 β(π,µ,st)<b and ρ z,t+k = min πt+i µz,t+i,ρ and c z,t+k are behaviour dependent clipped importance rations. Thus both G π,µz IS and G Vtrace are a multi-step return estimators with adaptive length. Note that only estimators with length ≥ 1 are used in V π trusted. Due to Minkowski's inequality the trust region estimator thus shows at least the same contraction as a 1-step bootstrap, but can be faster due to its adaptive nature: be a set of importance sampling estimators as defined in equation 7. Note that they all have the same fix point V π and contract with at least γ. Then the contraction properties carry over to V π trusted. In particular Proof. See Appendix C. Vtrace be a set of V-trace estimators (see equation 8) with corresponding fixed points V z (see equation 3) to which they contract at a speed of an algorithm and behaviour specific Proof. See Appendix C. Note how the choice of β and thus M β,π enables us to discard ill-suited G π,µz Vtrace from the estimation of V π trusted. Recall that V-trace fixed points V z are biased. Thus β allows us to selectively create the V-trace target and control its bias and the shrinkage Similarly it can control cases where we can not use the exact importance sampled estimator. The same approach based on nested expectations can be applied to the expectation of the policy gradient estimate and allows to control the bias and greediness (see Proposition 2) there as well. In Proposition 5 we have seen that the quality of the trust region V-trace return estimator depends on β. A suitable choice of β can move the return estimate V β closer to V π and improve the shrinkage by Hence, we employ a behaviour relevance function β KL that rejects high bias transitions by estimating the Kulback-Leibler divergence between the target policy π and the implied policyπ µz for a sampled behaviour µ z. Recall from Proposition 1 thatπ µz determines the fixed point of the V-trace estimator for behaviour µ z and thus determines the bias in V z. Note that the behaviour probabilities µ z can be evaluated and saved to the replay when the agent executes the behaviour, similarly the target policy π is represented by the agents neural network. Using both and equation 3,π µ can be computed. For large or infinite action spaces a Monte Carlo estimate of the KL divergence can be computed. It is possible to define separate behaviour relevance functions for the policy and value estimate. For simplicity we reject transitions entirely for all estimates and do not consider rejected transitions for the policy gradient and value gradient updates or auxiliary tasks. As described above we stop the Monte-Carlo bootstraps once they reach undesirable state-behaviour pairs. Note that this censoring procedure is computed from state dependent β(π, µ, s) and ensures that the choice of bootstrapping does not depend on the sampled actions. Note that rejection by an action-based criteria such as small π(a|s)/µ(a|s) would introduce an additional bias which we avoid by choosing β KL. We present experiments to support the following claims: • Section 5.2: Uniform experience replay obtains comparable as prioritized experience replay, while being simpler to implement and tune. • Section 5.3: Using fresh experience before inserting it in experience replay is better than learning purely off-policy from experience replay -in line with Proposition 3. • Section 5.4: Sharing experience without trust region performs poorly as suggested by Proposition 2. Off-Policy Trust-Region V-trace solves this issue. • Section 5.5: Sharing experience can take advantage of parallel exploration and obtains state-of-the-art performance on Atari games, while also saving memory through sharing a single experience replay. We use the V-trace distributed reinforcement learning agent . Updates are computed on mini-batches of 32 (regular) and 128 (replay) trajectories, each corresponding to 19 steps in the environment. In the context of DeepMind Lab, we consider the multi-task suite DMLab-30 , as the visuals and the dynamics are more consistent across tasks. Furthermore the multi-task regime is particularly suitable for the investigation of strongly off-policy data distributions arising from sharing the replay across agents, as concurrently learning agents can easily be stuck in different policy plateaus, generating substantially different data . As in , in the multi-task setting each agent trains simultaneously on a uniform mixture of all tasks rather than individually on each game. The score of an agent is thus the median across all 30 tasks. , we augment our agent with multi-task Pop-Art normalization and PixelControl. We use a PreCo LSTM instead of the vanilla one . Updates are computed on mini-batches of multiple trajectories chosen as above, each corresponding to 79 steps in the environment. In early experiments we found that computing the entropy cost only on the online data provided slightly better , hence we have done so throughout our experiments. In all our experiments, experience sampled from memory is mixed with online data within each minibatch -following Proposition 3. Episodes are removed in a first in first out order, so that replay always holds the most recent experience. Unless explicitly stated otherwise we consider hyper-parameter sweeps, some of which share experience via replay. In this setting multiple agents start from-scratch, run concurrently at identical speed, and add their new experience into a common replay buffer. All agents will then draw uniform samples from the replay buffer. On DMLab-30 we consider both regular hyper-parameter sweeps and sweeps with population based training (PBT) (a). On DMLab-30 sweeps contain 10 agents with hyper-parameters sampled similar as but fixed RMSProp = 0.1. On Atari sweeps contain 9 agents with different constant learning rate and entropy cost combinations {3 · 10 −4, 6 · 10 −4, 1.2 · 10 −3} × {5 · 10 −3, 1 · 10 −2, 2 · 10 −2} (distributed by factors {1/2, 1, 2} around the initial parameters reported in ). Although our focus is on efficient hyper-parameter sweeps given crude initial parameters, we also present a single-agent LASER experiment using the same tuned schedule as , a 87.5% replay ratio and a 15M replay. We store the entire episodes in the replay buffer and replay each episode from the beginning, using the most recent network parameters to recompute the LSTM states along the way: this is particularly critical when sharing experience between different agents, which may have arbitrarily different state representations. Prioritized experience replay has the potential to provide more efficient learning compared to uniform experience replay (; . However, it also introduces a number of new hyper-parameters and design choices: the most critical are the priority metric, how strongly to bias the sampling distribution, and how to correct for the ing bias. Uniform replay is instead almost parameter-free, requires little tuning and can be easily shared between multiple agents. Experiments provided in Figure 4 in the appendix showed little benefit of actor critic prioritized replay on DMLab-30. Furthermore priorities are typically computed from the agent specific metrics such as the TD-error, which are ill-defined when replay is shared among multiple agents. Hence we used uniform replay for our further investigations. Figure 2 (left) shows that performance degrades significantly when online data is not present in the batch. This experimentally validates Propositions 2 and 3 that highlight difficulties of learning purely off-policy. Furthermore Figure 2 (right) shows that best are obtained with experience replay of 10M capacity and 87.5% ratio. A ratio of 87.5% = 7/8 corresponds to 7 replay samples for each online sample. We have considered ratios of 1/2, 3/4, and 7/8 and observed stable training for all of them. Observe that among those values, larger ratios are more data-efficient as they take advantage of more replayed experience per training step. In line with proposition 2 we observe in Figure 3 (left) that hyper-parameter sweeps without trustregion are even surpassed by the baseline without experience replay. State-of-the-art are obtained in Figure 3 (right) when experience is shared with trust-region in a PBT sweep. Observe that this indicates parallel exploration benefits and saves memory at the same time: in our sweep of 10 replay agents the difference between 10 × 10M (separate replays) and 10M (shared replay) is 10-fold. This effect would be even more pronounced with larger sweeps. As discussed in section 2.3, the bias in V-trace occurs due to the clipping of importance ratios. A potential solution of reducing the bias would be to increase theρ threshold to clip less aggressively and accept increased variance. Figure 4 in the appendix shows that this is not a solution. We apply our proposed agent to Atari which has been a long established suite to evaluate reinforcement algorithms . Since we focus on sample-efficient learning we present our in comparison to prior work at 200M steps (Figure 1). Shared experience replay obtains even better performance than not shared experience replay. This confirms the efficient use of parallel exploration . The fastest prior agent to reach 400% is presented by requiring more than 3,000M steps. LASER with shared replay achieves 423% in 60M per agent. Given 200M steps it achieves 448%. We also present a single (no sweep) LASER agent that achieves 431% in 200M steps. We have presented LASER -an off-policy actor-critic agent which employs a large and shared experience replay to achieve data-efficiency. By sharing experience between concurrently running experiments in a hyper-parameter sweep it is able to take advantage of parallel exploration. As a it achieves state-of-the-art data efficiency on 57 Atari games given 200M environment steps. Furthermore it achieves competitive on both DMLab-30 and Atari under regular, not shared experience replay conditions. To facilitate this algorithm we have proposed two approaches: a) mixing replayed experience and on-policy data and b) a trust region scheme. We have shown theoretically and demonstrated through a series of experiments that they enable learning in strongly off-policy settings, which present a challenge for conventional importance sampling schemes. Increasing the clipping constantρ in V-trace reduces bias in favour of increased variance. We investigate if reducing bias in this manner enables sharing experience replay between multiple agents in a hyper-parameter sweep. Figure 4 (left) shows that this is not a solution, thus motivating our trust region scheme. In fact sharing experience replay in this particular way is worse than pure online learning. This motivates the use of our proposed trust region scheme. On a side note, increased clipping thresholds ing in worse performance verifies the importance of variance reduction through clipping. Right: Median human normalized performance across 30 tasks for the best agent in a sweep, averaged across 2 replicas. All replay experiments use 50% replay ratio and a capacity of 3 million observations. We investigate if uncorrected LSTM states can be used in combination with different replay modes. We consider uniform sampling and prioritization via the critic's loss, and include both full (β = 1) and partial (β = 0.5) importance corrections A.2 PRIORITIZED AND UNIFORM EXPERIENCE REPLAY, LSTM STATES With prioritized experience replay each transition τ is sampled with probability P (τ) ∝ p α τ, for a suitable unnormalized priority score p τ and a global tunable parameter α. It is common to then weight updates computed from that sample by 1/P (τ) β for 0 < β ≤ 1, where β = 1 fully corrects for the bias introduced in the state distribution. In one step temporal difference methods, typical priorities are based on the immediate TD-error, and are typically recomputed after a transition is sampled from replay. This means low priorities might stay low and get stale -even if the transition suddenly becomes relevant. To alleviate this issue, the sampling distribution is mixed with a uniform, as controlled by a third hyper parameter. The performance of agents with prioritized experience replay can be quite sensitive to the hyperparameters α, β, and. A critical practical consideration is how to implement random access for recurrent memory agents such as agents using an LSTM. Prioritized agents sample a presumably interesting transition from the past. This transition may be at any position within the episode. To infer the correct recurrent memory-state at this environment-state all earlier environment-states within that episode would need to be replayed. A prioritized agent with a random access pattern would thus require costly LSTM refreshes for each sampled transition. If LSTM states are not recomputed representational missmatch occurs. Sharing experience between multiple agents amplifies the issue of LSTM state representation missmatch. Here each agent has its own network parameters and the state representations between agents may be arbitrarily different. As a mitigation use a burn-in window or to initialize with a constant starting state. We note that those solutions can only partially mitigate the fundamental issue and that counter examples such as arbitrarily long T-Mazes can be constructed easily. We thus advocate for uniform sampling. In our implementation we uniformly sample an episode. Then we replay each episode from the beginning, using the most recent network parameters to recompute the LSTM states along the way: this is particularly critical when sharing experience between different agents, which may have arbitrarily different state representations. This solution is exact and cost-efficient as it only requires one additional forward pass for each learning step (forward + backward pass). An even more cost efficient approach would be to not refresh LSTM states at all. Naturally this comes at the cost of representational missmatch. However it would allow for an affordable implementation of prioritized experience replay. We investigate this in Figure 4 (right) and observe that it is not viable. We compare a baseline V-trace agent with no experience replay, one with uniform experience replay, and two different prioritized replay agents. We do not refresh LSTM states for any of the agents. The uniform replay agent is more data efficient then the baseline, and also saturates at a higher level of performance. The best prioritized replay agent uses full importance sampling corrections (β = 1). However it performs no higher than with uniform replay. We therefore we used uniform replay with full state correction for all our investigations in the paper. For evaluation, we average episode returns within buckets of 1M (Atari) and 10M (DMLab) environment steps for each agent instance, and normalize scores on each game by using the scores of a human expert and a random agent (van). In the multi-task setting, we then define the performance of each agent as the median normalized score of all levels that the agent trains on. Given the use of population based training, we need to perform the comparisons between algorithms at the level of sweeps. We do so by selecting the best performing agent instance within each sweep at any time. Note that for the multi-task setting, our approach of first averaging across many episodes, then taking the median across games, on DMLab further downsampling to 100M env steps, and only finally selecting the maximum within the sweep, in substantially lower variance than if we were to compute the maximum before the median and smoothing. All DMLab-30 sweeps are repeated 3× with the exception of ρ = 2 and ρ = 4 in Figure 4. We then plot a shaded area between the point-wise best and worst replica and a solid line for the mean. Atari sweeps having 57 games are summarized and plotted by the median of the human-normalized scores. We present algorithm pseudocode for LASER with trust region (Algorithm 1). For clarity we present a version without LSTM and focus on the single agent case. The multi-agent case is a simple extension where all agents save to the same replay database and also sample from the same replay. Also each agent starts with different network parameters and hyper-parameters. The LSTM state recomputation can be achieved with Replayer Threads (nearly identical to Actor Threads) that sample entire epsiodes from replay, step through them while reevaluating the LSTM state and slice the experience into trajectories of length T. Similar to regular LSTM Actor Threads from the Replayer Threads send each trajectory together with an LSTM state to the learning thread via a queue. The Learner Thread initializes the LSTM with the transmitted state when the LSTM is unrolled over the trajectory. Initialize parameter vectors θ. Initialize π 1 = π θ. Actor Thread: while training is ongoing do Sample trajectory unroll u = {τ t} t∈{1,...,T} of length T by acting in the environment using the latest π k where τ t = (s t, a t, r t, µ t = π k (s t |·)). Compute trust-region V-trace return V t,b using 8 where, where ρ is the clipped v-trace importance sampling ratio. Perform gradient update to θ using, denote the ing π θ as π k+1. end for We have stated five propositions in our paper for which we provide proofs below. Proposition 1. The V-trace value estimate Vπ is biased: It does not match the expected return of π but the return of a related implied policyπ defined by equation 9 that depends on the behaviour policy µ:π Proof.. Proposition 2. The V-trace policy gradient is biased: given the the optimal value function V * the V-trace policy gradient does not converge to a locally optimal π * for all off-policy behaviour distributions µ. Consider a tabular counter example with a single (locally) optimal policy at s t given by π * (s t) = argmax π a∈A π(a|s t)Q * (a, s t) that always selects the action argmax a Q * (a, s t). Even in this ideal tabular setting V-trace policy gradient estimates a differentπ * rather than the optimal π * as follows ∇V *,π (s t) = E µ [ρ t (r t + γV * (s t+1)∇ log π(a t |s t)] = E µ [ρ t Q * (s t, a t)∇ log π(a t |s t)] = E µ min π(a t |s t) µ(a t |s t),ρ Q * (s t, a t)∇ log π(a t |s t) = E µ π(a t |s t) µ(a t |s t) min 1,ρ µ(a t |s t) π(a t |s t) Q * (s t, a t)∇ log π(a t |s t) = E π min 1,ρ µ(a t |s t) π(a t |s t) Q * (s t, a t)∇ log π(a t |s t) = E π [ω(s t, a t)Q * (s t, a t)∇ log π(a t |s t)] = E π [Q *,ω (s t, a t)∇ log π(a t |s t)] Observe how the optimal Q-function Q * is scaled by ω(s t, a t) = min 1,ρ µ(at|st) π(at|st) ≤ 1 ing in implied state-action values Q *,ω. This penalizes actions where µ(a t |s t)ρ < π(a t |s t) and makes V-trace greedy w.r.t. to the remaining ones. Thus µ can be chosen adversarially to corrupt the optimal state action value. Note thatρ is a constant typically chosen to be 1. To prove the lemma consider a counter example such as an MDP with two actions and Q * = and µ = (0.9, 0.1) and initial π = (0.5, 0.5). Here the second action with expected return 5 is clearly favourable. Abusing notation µ/π = (1.8, 0.2). Thus Qπ,ω = (2 * 1, 5 * 0.2) =. Thereforẽ π * = wrongly selects the first action. with the V-trace distortion factor ω(s t, a t) = min 1,ρ µ(at|st) π(at|st) ≤ 1 that can de-emphasize action values and Q ω (s, a) = ω(s, a)Q(s, a). We display the Atari per-level performance of various agents at 50M and 200M environment steps in Table 2. The scores correspond to the agents presented in Figure 1. The LASER scores are computed by averaging the last 100 episode returns before 50M or respectively 200M environment frames have been experienced. Following the procedure defined by we initialize the environment with a random number of no-op actions (up to 37 in our case). Again following episodes are terminated after 30 minutes of gameplay. Note that have not published per-level scores. Rainbow scores are obtained from.

Represent the short summary for finding the corresponding scientific article (long)

We investigate and propose solutions for two challenges in reinforcement learning: (a) efficient actor-critic learning with experience replay (b) stability of very off-policy learning.

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the scientific article to find a short TLDR

Question: Stochastic gradient descent (SGD) is the workhorse of modern machine learning. Sometimes, there are many different potential gradient estimators that can be used. When so, choosing the one with the best tradeoff between cost and variance is important. This paper analyzes the convergence rates of SGD as a function of time, rather than iterations. This in a simple rule to select the estimator that leads to the best optimization convergence guarantee. This choice is the same for different variants of SGD, and with different assumptions about the objective (e.g. convexity or smoothness). Inspired by this principle, we propose a technique to automatically select an estimator when a finite pool of estimators is given. Then, we extend to infinite pools of estimators, where each one is indexed by control variate weights. This is enabled by a reduction to a mixed-integer quadratic program. Empirically, automatically choosing an estimator performs comparably to the best estimator chosen with hindsight. In stochastic gradient variational inference (SGVI) there are multiple gradient estimators with varying costs and variances. Estimators may be obtained using the reparameterization trick (; ; Titsias and Lázaro-), the score function method , or other techniques (Titsias and Lázaro- ; ;). Also, many control variates can be added to an estimator to reduce variance . The cost and variance of an estimator significantly affects optimization convergence speed . The use of different estimators leads to different optimization performances, and the estimator with optimal cost-variance tradeoff is often situationdependent (for an example see Fig. 1). In settings where multiple estimators with varying costs and variances are available, selecting the optimal one is important. Rather than rely on the user to manually select one, we propose that estimator selection could be done adaptively. This paper investigates how, given a pool of gradient estimators, automatically choose one to get the best convergence guarantee for stochastic optimization. We study cost-variance tradeoffs by analyzing the convergence rates of several variants of SGD. We express convergence rates in terms of time rather than iterations. This leads to what we call the "G 2 T principle": A simple rule that predicts, given a pool of gradient estimators, which one in the best convergence guarantees for optimization. We use the principle to propose two gradient estimator selection algorithms: One for the case in which a finite pool of estimators is available, and other when the pool contains an infinite number of estimators, each indexed by control variate weights (i.e. control variate selection). Notation: We use g(w, ξ), where ξ is a random variable, to denote an unbiased estimator of target's gradient, G 2 (g) to denote a bound on g's expected squared norm, and T (g) to denote the computational cost of computing estimator g(w, ξ), measured in seconds. 2 T Principle and Gradient Estimator Selection Given a set of gradient estimators with varying costs and variances, our goal is to find the one that gives the best convergence guarantee for optimization algorithms. Convergence guarantees for several variants of SGD are shown in Table 1. Given a pool of estimators with known G 2 and T, the one with minimum G 2 T should be used. In practice, however, G 2 and T are typically not known. We propose to use estimates. Assuming that the cost of an estimator g(w, ξ) is independent of w, an estimateT (g) of T (g) can be obtained for each g ∈ G through a single initial profiling phase. Dealing with G 2 (g) is more involved. Convergence guarantees assume that E ||g(w, ξ)|| 2 ≤ G 2 (g) for all w. Often (e.g. when w is unbounded) this is not true for any finite G. We propose an approach that is justified under two assumptions: (i) Optimization starting from a point w 0 will only visit a restricted part of parameter space. It is sufficient to bound E ||g(w, ξ)|| 2 for the set of w that may actually be encountered. (ii) E g(w, ξ) 2 tends to decrease slowly over time. When these are true, it makes sense to also form an estimatê G 2 (g) through an initial profiling phase, and to update these estimates a small number of times as optimization proceeds. The approach is summarized in Alg. 1. It is possible to use multiple control variates to reduce a gradient estimator's variance. However, some control variates might be computationally expensive but only in a small reduction in variance. It may be better to remove them and accept a noisier but cheaper estimator. How can we select what control variates to use? When an unbiased gradient estimator g base and control variates c 1,...c J are given, a gradient estimator can be expressed as The available estimators are G = {g a : a ∈ R J}. The number of estimators g a ∈ G is infinite, and Alg. 1 cannot be used (cannot measureT andĜ 2 for each estimator g a individually). We show that, despite having an infinite number of estimators, when estimators are indexed by control variate weights finding the one with minimumĜ 2T can be done efficiently. This is because two properties hold: (i) EstimatesT (g a) andĜ 2 (g a) can be efficiently obtained for all estimators g a ∈ G through the use of shared statistics (a finite number of evaluations of the base estimator and control variates); and (ii) The ing (combinatorial) optimization problem a * = arg min aĜ 2 (g a)T (g a) can be reduced to a Mixed Integer Quadratically Constrained Program (MIQCP), which can be solved quickly in practice. For all g ∈ G measure timeT (g). for k = 1, 2, · · · do if time to re-select estimator then for each estimator g dô Algorithm 2 SGD with automatically selected control variates. Require: Set of estimators, G. Require: Times to re-select estimator. Require: Number of MC samples M. For g base measure time t 0. This section presents an overview of the experiments. Full details are in the appendix. We tackle inference problems using SGVI. We consider three models: Logistic regression, a hierarchical regression model, and a Bayesian neural network. For the simple logistic regression model we use a Gaussian with a full rank covariance as variational distribution q w (z). For the other more complex models we use a factorized Gaussian. We use SGD with momentum to optimize, and five samples z ∼ q w (z) to form Monte Carlo gradient estimates. For both Algs. 1 and 2 we update the estimator used (by minimizingĜ 2T) three times during training. We first present an empirical validation for Alg. 1. We compare the achieved by using three different gradient estimators: (Rep) the plain reparameterization estimator, (Miller) the estimator proposed by Miller et al. , and (STL) the "sticking the landing" estimator . We also run Alg. 1 with the set of estimators G = {Rep, Miller, STL}, which uses the estimator g ∈ G with minimumĜ 2T. We now present an empirical validation for Alg. 2 (control variate selection). We consider the same three models as above. The set of candidate estimators is G Auto = {g a : a ∈ R 3}, where g a is as defined in eq.. The base estimator is plain reparameterization, and there are three candidate control variates (c 1, c 2, c 3). The goal is to check if Alg. 2 successfully navigates cost/variance tradeoffs. We thus compare against using each possible fixed subsets of control variates S ⊆ {c 1, c 2, c 3}, with the weights that minimize the estimator's variance (which can be estimated efficiently ( Log. Regression (a1a) We compare against using different fixed subsets of control variates with the weights that minimize the estimator's variance. Lines are identified as follows: "Auto" stands for using Alg. 2 to select what control variates to use and their weights, "Base" stands for optimizing using the base gradient alone, "1" stands for using the fixed set of control variates {c 1} with the minimum variance weights, "12" stands for using the fixed set of control variates {c 1, c 2}, and so on. Appendix A. Appendix Three different models were considered: a Bayesian neural network, a hierarchical Poisson model, and Bayesian logistic regression. Bayesian logistic regression: We use the dataset a1a. The training set is given by, where y i is binary. The model is specified by Hierarchical Poisson model:. The model measures the relative stop-and-frisk events in different precincts in New York city, for different ethnicities. The model is specified by In this case, e stands for ethnicity and p for precinct, Y ep for the number of stops in precinct p within ethnicity group e (observed), and N ep for the total number of arrests in precinct p within ethnicity group e (observed). BNN: As done by Miller et al. we use a subset of 100 rows from the "Red-wine" dataset (regression). We implement a neural network with one hidden layer with 50 units and Relu activations. Let D = {x i, y i} N i=1 be the training set. The model is specified by log α ∼ N (0, 10 2), log τ ∼ N (0, 10 2), W ∼ N (0, α 2 I), (weights and biases) A.2. Details on the simulations Control variates used: c 1: Difference between the entropy term computed exactly and estimated using reparameterization: c(w, ξ) = ∇ w log q w (T w (ξ)) − ∇ w Eq w log q w (Z). c 2: Control variate by Miller et al. based on a second order Taylor expansion of log p(x, z). c 3: Difference between the prior term computed exactly and estimated using reparameterization: c(w, ξ) = ∇ w log p(T w (ξ)) − ∇ w Eq w log p(Z). Algorithmic details: For Alg. 2 we use M = 400 to estimateĜ 2 (except for Logistic regression, where we use M = 200). We re-select the optimal estimator three times during training, initially, after 10% of training is done, and after 50% of training is done. Optimization details: We use SGD with momentum (β = 0.9) with 5 samples z ∼ q w (z) to form the Monte Carlo gradient estimates. For all models we find an initial set of parameters by optimizing with the base gradient for 300 steps and a fixed learning rate of 10 −5. This initialization was helpful in practice because w tends to change rapidly at the beginning of optimization. After this brief initialization, E ||g(w, ξ)|| 2 tends to change much more slowly, meaning our technique is more helpful. The performance of all algorithms depends on the step-size. To give a fair comparison, Figs. 1 and 2 summarize by showing the with the best step-size for each estimator. (12 stepsizes between 10 −6 and 10 −3 were considered.) Estimators with control variates can be expressed as An expression forT (g a) can be obtained by noticing that computing g a only requires computing the base gradient and the control variates with non-zero weights. Then, for all 2. Using Tw(ξ) = µ + D 1/2 ξ, where ξ ∼ N (0, I), µ is the mean of qw and D 1/2 is the Cholesky factorization of the covariance of qw. Thus, we can computeT (g a) for all g a ∈ G only by profiling the base gradient and each control variate individually. Similarly,Ĝ 2 (g a, w) is determined by the same set of base gradient and control variate evaluations, regardless of the value of a. Suppose that, at iteration k, we sample ξ k1,..., ξ kM. Then, for all g a ∈ G, Thus, we can computeĜ 2 (g a, w k) for all g a ∈ G using only M evaluations of the base gradient g base and each control variate c i. Equations and characterize the (estimated) cost and variance of the gradient estimator with weights a. We find the weights that in the optimal cost-variance tradeoff by solving a * (w) = arg min whereT (g a) andĜ 2 (g a, w) are as in equations and. The solution a * (w) indicates what control variates to use (those with a * i = 0), and their weights. Solving the (combinatorial) minimization problem in equation may be challenging. However, theorem 1 states that it can be reduced to a MIQCP, which can be solved fast using solvers such as. Theorem 1 When different gradient estimators are indexed by a set of J control variate weights, the problem of finding a * (w) as in equation can be reduced to solving a mixed integer quadratically constrained program with 2J + 2 variables, one quadratic constraint, and one linear constraint. A mixed integer quadratic program is an optimization problem in which the objective function and constraints are quadratic (or linear), and some (or all) variables are restricted to be integers: Ax + b = 0, where x ∈ R n, Q 0,..., Q m ∈ R n×n, and some components of x are restricted to be integers. We now prove the theorem 1. Proof GivenT The final minimization problem shown in equation 14 has the form of a general MIQCP, shown in equation 6, with the exception of the last constraint b i = 1[a i = 0]. Despite not being in the original definition of a MIQCP, several solver accept constraints of this type (, the solver used in our simulation, does).

Represent the TLDR for finding the corresponding scientific article

We propose a gradient estimator selection algorithm with the aim on improving optimization efficiency.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design. The discrepancy between the minimax and maximin objective values could serve as a proxy for the difficulties that the alternating gradient descent encounters in the optimization of GANs. In this work, we give new on the benefits of multi-generator architecture of GANs. We show that the minimax gap shrinks to \epsilon as the number of generators increases with rate O(1/\epsilon). This improves over the best-known of O(1/\epsilon^2). At the core of our techniques is a novel application of Shapley-Folkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constraint optimization problem. Our proposed Stackelberg GAN performs well experimentally in both synthetic and real-world datasets, improving Frechet Inception Distance by 14.61% over the previous multi-generator GANs on the benchmark datasets. Generative Adversarial Nets (GANs) are emerging objects of study in machine learning, computer vision, natural language processing, and many other domains. In machine learning, study of such a framework has led to significant advances in adversarial defenses BID25 BID22 and machine security BID3 BID22. In computer vision and natural language processing, GANs have ed in improved performance over standard generative models for images and texts BID11, such as variational autoencoder BID14 and deep Boltzmann machine BID20. A main technique to achieve this goal is to play a minimax two-player game between generator and discriminator under the design that the generator tries to confuse the discriminator with its generated contents and the discriminator tries to distinguish real images/texts from what the generator creates. Despite a large amount of variants of GANs, many fundamental questions remain unresolved. One of the long-standing challenges is designing universal, easy-to-implement architectures that alleviate the instability issue of GANs training. Ideally, GANs are supposed to solve the minimax optimization problem BID11, but in practice alternating gradient descent methods do not clearly privilege minimax over maximin or vice versa (page 35,), which may lead to instability in training if there exists a large discrepancy between the minimax and maximin objective values. The focus of this work is on improving the stability of such minimax game in the training process of GANs. To alleviate the issues caused by the large minimax gap, our study is motivated by the so-called Stackelberg competition in the domain of game theory. In the Stackelberg leadership model, the players of this game are one leader and multiple followers, where the leader firm moves first and then the follower firms move sequentially. It is known that the Stackelberg model can be solved to find a subgame perfect Nash equilibrium. We apply this idea of Stackelberg leadership model to the architecture design of GANs. That is, we design an improved GAN architecture with multiple generators (followers) which team up to play against the discriminator (leader). We therefore name our model Stackelberg GAN. Our theoretical and experimental establish that: GANs with multi-generator architecture have smaller minimax gap, and enjoy more stable training performances. Our Contributions. This paper tackles the problem of instability during the GAN training procedure with both theoretical and experimental . We study this problem by new architecture design. Figure 1: Stackelberg GAN stabilizes the training procedure on a toy 2D mixture of 8 Gaussians. Top Row: Standard GAN training. It shows that several modes are dropped. Bottom Row: Stackelberg GAN training with 8 generator ensembles, each of which is denoted by one color. We can see that each generator exactly learns one mode of the distribution without any mode being dropped. Under review as a conference paper at ICLR 2019 (a) Step 0 Standard GAN training. It shows that several modes are dropped. Bottom Row: Stackelberg GAN training with 8 generator ensembles, each of which is denoted by one color. We can see that each generator exactly learns one mode of the distribution without any mode being dropped.• We propose Stackelberg GAN framework of having multiple generators in the GAN architecture. Our framework is general that can be applied to all variants of GANs, e.g., vanilla GAN, Wasserstein GAN, etc. It is built upon the idea of jointly optimizing an ensemble of GAN losses w.r.t. all pairs of discriminator and generator. Differences with prior work. Although the idea of having multiple generators in the GAN architecture is not totally new, e.g., MIX+GAN BID1 and MGAN BID13, there are key differences between Stackelberg GAN and prior work. a) In MGAN BID13, various generators are combined as a mixture of probabilistic models with assumption that the generators and discriminator have enough capacity. In contrast, in the Stackelberg GAN model we uniformly ensemble the losses of various standard GAN without any assumption on the model capacity. b) In MIX+GAN BID1, the losses are ensembled with learned weights and an extra regularization term, which discourages the weights being too far away from uniform. We find it slightly unnecessary because the expressive power of each generator already allows implicit scaling of each generator. To the contrary, in the Stackelberg GAN we apply equal weights for all generators.• We prove that the minimax duality gap shrinks as the number of generators increases (see Theorem 1 and Corollary 2). Unlike the previous work, our has no assumption on the expressive power of generators and discriminator, but instead depends on their non-convexity. With extra condition on the expressive power of generators, we show that Stackelberg GAN is able to achieve ✏-approximate equilibrium with e O(1/✏) generators (see Theorem 3). This Stackelberg GAN training with 10 generator ensembles on real images without cherry pick, where each row corresponds to one generator. We can see that each generator exactly learns one mode of the distribution without any mode being dropped.[Pengtao: It is kind of abrupt that you say "Stackelberg GAN stabilizes the training procedure" in the beginning sentence, then the rest talks about losing mode. In the introduction, a convincing tie between instability and mode collapse is still missing.]• We propose Stackelberg GAN framework of having multiple generators in the GAN architecture. Our framework is general that can be applied to all variants of GANs, e.g., vanilla GAN, Wasserstein GAN, etc. It is built upon the idea of jointly optimizing an ensemble of GAN losses w.r.t. all pairs of discriminator and generator. Differences with prior work. Although the idea of having multiple generators in the GAN architecture is not totally new, e.g., MIX+GAN BID1 and MGAN BID13, there are key differences between Stackelberg GAN and prior work. a) In MGAN BID13, various generators are combined as a mixture of probabilistic models with assumption that the generators and discriminator have enough capacity. In contrast, in the Stackelberg GAN model we uniformly ensemble the losses of various standard GAN without any assumption on the model capacity. b) In MIX+GAN BID1, the losses are ensembled with learned weights and an extra regularization term, which discourages the weights being too far away from uniform. We find it slightly unnecessary because the expressive power of each generator already allows implicit scaling of each generator. To the contrary, in the Stackelberg GAN we apply equal weights for all generators.• We prove that the minimax duality gap shrinks as the number of generators increases (see Theorem 1 and Corollary 2). Unlike the previous work, our has no assumption on the • We propose the Stackelberg GAN framework of multiple generators in the GAN architecture. Our framework is general since it can be applied to all variants of GANs, e.g., vanilla GAN, Wasserstein GAN, etc. It is built upon the idea of jointly optimizing an ensemble of GAN losses w.r.t. all pairs of discriminator and generator. Differences from prior work. Although the idea of having multiple generators in the GAN architecture is not totally new, e.g., MIX+GAN BID1, MGAN BID13, MAD-GAN BID9 and GMAN BID8, there are key differences between Stackelberg GAN and prior work. a) In MGAN BID13 and MAD-GAN BID9, various generators are combined as a mixture of probabilistic models with assumption that the generators and discriminator have infinite capacity. Also, they require that the generators share common network parameters. In contrast, in the Stackelberg GAN model we allow various sampling schemes beyond the mixture model, e.g., each generator samples a fixed but unequal number of data points independently. Furthermore, each generator has free parameters. We also make no assumption on the model capacity in our analysis. This is an important research question as raised by BID2. b) In MIX+GAN BID1, the losses are ensembled with learned weights and an extra regularization term, which discourages the weights being too far away from uniform. We find it slightly unnecessary because the expressive power of each generator already allows implicit scaling of each generator. In the Stackelberg GAN, we apply equal weights for all generators and obtain improved guarantees. c) In GMAN BID8, there are multiple discriminators while it is unclear in theory why multi-discriminator architecture works well. In this paper, we provide formal guarantees for our model. • We prove that the minimax duality gap shrinks as the number of generators increases (see Theorem 1 and Corollary 2). Unlike the previous work, our has no assumption on the expressive power of generators and discriminator, but instead depends on their non-convexity. With extra condition on the expressive power of generators, we show that Stackelberg GAN is able to achieve -approximate equilibrium with O(1/) generators (see Theorem 3). This improves over the best-known in BID1 which requires generators as many as O(1/ 2). At the core of our techniques is a novel application of the ShapleyFolkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constrained optimization problem. This in tighter bounds than that of the covering number argument as in BID1. We also note that MIX+GAN is a heuristic model which does not exactly match the theoretical analysis in BID1, while this paper provides formal guarantees for the exact model of Stackelberg GAN.• We empirically study the performance of Stackelberg GAN for various synthetic and real datasets. We observe that without any human assignment, surprisingly, each generator automatically learns balanced number of modes without any mode being dropped (see FIG2). Compared with other multi-generator GANs with the same network capacity, our experiments show that Stackelberg GAN enjoys 26.76 Fréchet Inception Distance on CIFAR-10 dataset while prior achieve 31.34 (smaller is better), achieving an improvement of 14.61%. Before proceeding, we define some notations and formalize our model setup in this section. Notations. We will use bold lower-case letter to represent vector and lower-case letter to represent scalar. Specifically, we denote by θ ∈ R t the parameter vector of discriminator and γ ∈ R g the parameter vector of generator. Let D θ (x) be the output probability of discriminator given input x, and let G γ (z) represent the generated vector given random input z. For any function f (u), we denote by f DISPLAYFORM0 } the conjugate function of f. Letclf be the convex closure of f, which is defined as the function whose epigraph is the convex closed hull of that of function f. We define clf:= −cl(−f). We will use I to represent the number of generators. Preliminaries. The key ingredient in the standard GAN is to play a zero-sum two-player game between a discriminator and a generator -which are often parametrized by deep neural networks in practice -such that the goal of the generator is to map random noise z to some plausible images/texts G γ (z) and the discriminator D θ (·) aims at distinguishing the real images/texts from what the generator creates. For every parameter implementations γ and θ of generator and discriminator, respectively, denote by the payoff value DISPLAYFORM0 where f (·) is some concave, increasing function. Hereby, P d is the distribution of true images/texts and P z is a noise distribution such as Gaussian or uniform distribution. The standard GAN thus solves the following saddle point problems: DISPLAYFORM1 For different choices of function f, problem leads to various variants of GAN. For example, when f (t) = log t, problem is the classic GAN; when f (t) = t, it reduces to the Wasserstein GAN. We refer interested readers to the paper of BID18 for more variants of GANs. Stackelberg GAN. Our model of Stackelberg GAN is inspired from the Stackelberg competition in the domain of game theory. Instead of playing a two-player game as in the standard GAN, in Stackelberg GAN there are I + 1 players with two firms -one discriminator and I generators. One can make an analogy between the discriminator (generators) in the Stackelberg GAN and the leader (followers) in the Stackelberg competition. Stackelberg GAN is a general framework which can be built on top of all variants of standard GANs. The objective function is simply an ensemble of losses w.r.t. all possible pairs of generators and discriminator: DISPLAYFORM2 Thus it is very easy to implement. The Stackelberg GAN therefore solves the following saddle point problems: DISPLAYFORM3 We term w * − q * the minimax (duality) gap. We note that there are key differences between the naïve ensembling model and ours. In the naïve ensembling model, one trains multiple GAN models independently and averages their outputs. In contrast, our Stackelberg GAN shares a unique discriminator for various generators, thus requires jointly training. FIG3 shows the architecture of our Stackelberg GAN.How to generate samples from Stackelberg GAN? In the Stackelberg GAN, we expect that each generator learns only a few modes. In order to generate a sample that may come from all modes, we use a mixed model. In particular, we generate a uniformly random value i from 1 to I and use the i-th generator to obtain a new sample. Note that this procedure in independent of the training procedure. In this section, we develop our theoretical contributions and compare our with the prior work. We begin with studying the minimax gap of Stackelberg GAN. Our main show that the minimax gap shrinks as the number of generators increases. To proceed, denote by DISPLAYFORM0, where the conjugate operation is w.r.t. the second argument of φ(γ i ; ·). We clarify here that the subscript i in h i indicates that the function h i is derived from the i-th generator. The argument of h i should depend on i, so we denote it by u i. Intuitively, h i serves as an approximate convexification of −φ(γ i, ·) w.r.t the second argument due to the conjugate operation. Denote byclh i the convex closure of h i: DISPLAYFORM1 clh i represents the convex relaxation of h i because the epigraph ofclh i is exactly the convex hull of epigraph of h i by the definition ofclh i. Let DISPLAYFORM2 measures the non-convexity of objective function w.r.t. argument θ. For example, it is equal to 0 if and only if φ(γ i ; θ) is concave and closed w.r.t. discriminator parameter θ. We have the following guarantees on the minimax gap of Stackelberg GAN. DISPLAYFORM3 Denote by t the number of parameters of discriminator, i.e., θ ∈ R t. Suppose that h i (·) is continuous and domh i is compact and convex. Then the duality gap can be bounded by DISPLAYFORM4 provided that the number of generators I > t+1 ∆ worst γ. Remark 1. Theorem 1 makes mild assumption on the continuity of loss and no assumption on the model capacity of discriminator and generators. The analysis instead depends on their nonconvexity as being parametrized by deep neural networks. In particular, ∆, we have 0 ≤ w * − q * ≤. The of Theorem 1 and Corollary 2 are independent of model capacity of generators and discriminator. When we make assumptions on the expressive power of generator as in BID1, we have the following guarantee on the existence of -approximate equilibrium. Theorem 3. Under the settings of Theorem 1, suppose that for any ξ > 0, there exists a generator G such that E x∼P d,z∼Pz G(z) − x 2 ≤ ξ. Let the discriminator and the generators be L-Lipschitz w.r.t. inputs and parameters, respectively. Then for any > 0, DISPLAYFORM0 and a discriminator D θ * such that for some value V ∈ R, DISPLAYFORM1 Related Work. While many efforts have been devoted to empirically investigating the performance of multi-generator GAN, little is known about how many generators are needed so as to achieve certain equilibrium guarantees. Probably the most relevant prior work to Theorem 3 is that of BID1. In particular, BID1 showed that there exist I = 100t 2 ∆ 2 generators and one discriminator such that -approximate equilibrium can be achieved, provided that for all x and any ξ > 0, there exists a generator G such that E z∼Pz G(z) − x 2 ≤ ξ. Hereby, ∆ is a global upper bound of function |f |, i.e., f ∈ [−∆, ∆]. In comparison, Theorem 3 improves over this in two aspects: a) the assumption on the expressive power of generators in BID1. Therefore, Theorem 3 requires much fewer generators than that of BID1. DISPLAYFORM2 In this section, we empirically investigate the effect of network architecture and capacity on the mode collapse/dropping issues for various multi-generator architecture designs. Hereby, the mode dropping refers to the phenomenon that generative models simply ignore some hard-to-represent modes of real distributions, and the mode collapse means that some modes of real distributions are "averaged" by generative models. For GAN, it is widely believed that the two issues are caused by the large gap between the minimax and maximin objective function values (see page 35, BID10). Our experiments verify that network capacity (change of width and depth) is not very crucial for resolving the mode collapse issue, though it can alleviate the mode dropping in certain senses. Instead, the choice of architecture of generators plays a key role. To visualize this discovery, we test the performance of varying architectures of GANs on a synthetic mixture of Gaussians dataset with 8 modes and 0.01 standard deviation. We observe the following phenomena: Naïvely increasing capacity of one-generator architecture does not alleviate mode collapse. It shows that the multi-generator architecture in the Stackelberg GAN effectively alleviates the mode collapse issue. Though naïvely increasing capacity of one-generator architecture alleviates mode dropping issue, for more challenging mode collapse issue, the effect is not obvious (see FIG9 . (b) show that increasing the model capacity can alleviate the mode dropping issue, though it does not alleviate the mode collapse issue. (c) Multi-generator architecture with even small capacity resolves the mode collapse issue. Stackelberg GAN outperforms multi-branch models. We compare performance of multi-branch GAN and Stackelberg GAN with objective functions: DISPLAYFORM0 Hereby, the multi-branch GAN has made use of extra information that the real distribution is Gaussian mixture model with probability distribution function DISPLAYFORM1, so that each γ i tries to fit one component. However, even this we observe that with same model capacity, Stackelberg GAN significantly outperforms multi-branch GAN (see FIG11 (a)(c)) even without access to the extra information. The performance of Stackelberg GAN is also better than multi-branch GAN of much larger capacity (see FIG11). Generators tend to learn balanced number of modes when they have same capacity. We observe that for varying number of generators, each generator in the Stackelberg GAN tends to learn equal number of modes when the modes are symmetric and every generator has same capacity (see FIG13). In this section, we verify our theoretical contributions by the experimental validation. We first show that Stackelberg GAN generates more diverse images on the MNIST dataset than classic GAN. We follow the standard preprocessing step that each pixel is normalized via subtracting it by 0.5 and dividing it by 0.5. The detailed network setups of discriminator and generators are in TAB4. Figure 6 shows the diversity of generated digits by Stackelberg GAN with varying number of generators. When there is only one generator, the digits are not very diverse with many repeated "1"'s and much fewer "2"'s. As the number of generators increases, the generated images tend to be more diverse. In particular, for 10-generator Stackelberg GAN, each generator is associated with one or two digits without any digit being missed. We also observe better performance by the Stackelberg GAN on the Fashion-MNIST dataset. Fashion-MNIST is a dataset which consists of 60,000 examples. Each example is a 28 × 28 grayscale image associating with a label from 10 classes. We follow the standard preprocessing step that each pixel is normalized via subtracting it by 0.5 and dividing it by 0.5. We specify the detailed network setups of discriminator and generators in TAB4. Figure 7 shows the diversity of generated fashions by Stackelberg GAN with varying number of generators. When there is only one generator, the generated images are not very diverse without Figure 6: Standard GAN vs. Stackelberg GAN on the MNIST dataset without cherry pick. Left Figure: Digits generated by the standard GAN. It shows that the standard GAN generates many "1"'s which are not very diverse. Middle Figure: Digits generated by the Stackelberg GAN with 5 generators, where every two rows correspond to one generator. Right Figure: Digits generated by the Stackelberg GAN with 10 generators, where each row corresponds to one generator. Figure 7: Generated samples by Stackelberg GAN on CIFAR-10 dataset without cherry pick. Left Figure: Examples generated by the standard GAN. It shows that the standard GAN fails to generate bags. Middle Figure: Examples generated by the Stackelberg GAN with 5 generators, where every two rows correspond to one generator. Right Figure: Examples generated by the Stackelberg GAN with 10 generators, where each row corresponds to one generator. any bags being found. As the number of generators increases, the generated images tend to be more diverse. In particular, for 10-generator Stackelberg GAN, each generator is associated with one class without any class being missed. We then implement Stackelberg GAN on the CIFAR-10 dataset. CIFAR-10 includes 60,000 32×32 training images, which fall into 10 classes BID15 ). The architecture of generators and discriminator follows the design of DCGAN in BID19. We train models with 5, 10, and 20 fixed-size generators. The show that the model with 10 generators performs the best. We also train 10-generator models where each generator has 2, 3 and 4 convolution layers. We find that the generator with 2 convolution layers, which is the most shallow one, performs the best. So we report the obtained from the model with 10 generators containing 2 convolution layers. FIG15 shows the samples produced by different generators. The samples are randomly drawn instead of being cherry-picked to demonstrate the quality of images generated by our model. For quantitative evaluation, we use Inception score and Fréchet Inception Distance (FID) to measure the difference between images generated by models and real images. Results of Inception Score. The Inception score measures the quality of a generated image and is correlated well with human's judgment BID21. We report the Inception score obtained by our Stackelberg GAN and other baseline methods in TAB1. For fair comparison, we only consider the baseline models which are completely unsupervised model and do not need any label information. Instead of directly using the reported Inception scores by original papers, we replicate the experiment of MGAN using the code, architectures and parameters reported by their original papers, and evaluate the scores based on the new experimental . TAB1 shows that our model achieves a score of 7.62 in CIFAR-10 dataset, which outperforms the state-of-the-art models. For fairness, we configure our Stackelberg GAN with the same capacity as MGAN, that is, the two models have comparative number of total parameters. When the capacity of our Stackelberg GAN is as small as DCGAN, our model improves over DCGAN significantly. Results of Fréchet Inception Distance. We then evaluate the performance of models on CIFAR-10 dataset using the Fréchet Inception Distance (FID), which better captures the similarity between generated images and real ones BID12. As TAB1 shows, under the same capacity as DCGAN, our model reduces the FID by 20.74%. Meanwhile, under the same capacity as MGAN, our model reduces the FID by 14.61%. This improvement further indicates that our Stackelberg GAN with multiple light-weight generators help improve the quality of the generated images. Real data 11.24 ± 0.16 -WGAN 3.82 ± 0.06 -MIX+WGAN BID1 4.04 ± 0.07 -Improved-GAN BID21 4.36 ± 0.04 -ALI BID7 5.34 ± 0.05 -BEGAN BID4 5.62 -MAGAN BID24 5.67 -GMAN BID8 6.00 ± 0.19 -DCGAN BID19 6.40 ± 0.05 37.7 Ours (capacity as DCGAN)7.02 ± 0.07 29.88 D2GAN BID17 7.15 ± 0.07 -MAD-GAN (our run) BID9 6.67 ± 0.07 34.10 MGAN (our run) BID13 7.52 ± 0.1 31.34 Ours (FIG2 7.62 ± 0.07 26.76 We also evaluate the performance of Stackelberg GAN on the Tiny ImageNet dataset. The Tiny ImageNet is a large image dataset, where each image is labelled to indicate the class of the object inside the image. We resize the figures down to 32 × 32 following the procedure described in BID6 . FIG15 shows the randomly picked samples generated by 10-generator Stackelberg GAN. Each row has samples generated from one generator. Since the types of some images in the Tiny ImageNet are also included in the CIFAR-10, we order the rows in the similar way as FIG15 . In this work, we tackle the problem of instability during GAN training procedure, which is caused by the huge gap between minimax and maximin objective values. The core of our techniques is a multi-generator architecture. We show that the minimax gap shrinks to as the number of generators increases with rate O(1/), when the maximization problem w.r.t. the discriminator is concave. This improves over the best-known of O(1/ 2). Experiments verify the effectiveness of our proposed methods. TAB5 is by the weak duality. Thus it suffices to prove the other side of the inequality. All notations in this section are defined in Section 3.1. We first show that DISPLAYFORM0 Denote by DISPLAYFORM1 We have the following lemma. Lemma 4. We have DISPLAYFORM2 Proof. By the definition of p, we have p = inf γ1,...,γ I ∈R g sup θ∈R t Φ(γ 1, ..., γ I ; θ). Since (clp)(·) is the convex closure of function p(·) (a.k.a. weak duality theorem), we have (clp) ≤ p. We now show that sup DISPLAYFORM3 Note that p(u) = inf γ1,...,γ I ∈R g p γ1,...,γ I (u), where p γ1,...,γ I (u) = sup θ∈R t {Φ(γ 1, ..., γ I ; θ) − u T θ} = (− Φ(γ 1, ..., γ I ; ·)) * (−u), and that. We have the following lemma. DISPLAYFORM4 Lemma 5. Under the assumption in Theorem 1, DISPLAYFORM5 Proof. We note that DISPLAYFORM6 where u 1,..., u I, u ∈ R t. Therefore, DISPLAYFORM7 Consider the subset of R t+1: DISPLAYFORM8 Define the vector summation DISPLAYFORM9 is continuous and domh i is compact, the set DISPLAYFORM10 DISPLAYFORM11 We apply Lemma 6 to prove Lemma 5 with m = t + 1. Let (r, w) ∈ conv(Y) be such that r = 0, and w =clp. DISPLAYFORM12 i ∈I DISPLAYFORM13 Representing elements of the convex hull of DISPLAYFORM14 by Carathéodory theorem, we have that for each i ∈ I, there are vectors {u DISPLAYFORM15 Recall that we definȇ DISPLAYFORM16 and DISPLAYFORM17 We have for i ∈ I, DISPLAYFORM18 Thus, by Eqns. FORMULA27 and FORMULA30, we have DISPLAYFORM19 Therefore, we have DISPLAYFORM20 (by Eqns. FORMULA28 and FORMULA33) DISPLAYFORM21, (by Lemma 6) as desired. By Lemmas 4 and 5, we have proved that DISPLAYFORM22 To prove Theorem 1, we note that DISPLAYFORM23 When φ(γ i ; θ) is concave and closed w.r.t. discriminator parameter θ, we have clφ = φ. Thus, ∆ minimax θ = ∆ maximin θ = 0 and 0 ≤ w * − q * ≤. We first show that the equilibrium value V is 2f (1/2). For the discriminator D θ which only outputs 1/2, it has payoff 2f (1/2) for all possible implementations of generators G γ1,..., G γ I. Therefore, we have V ≥ 2f (1/2). We now show that V ≤ 2f (1/2). We note that by assumption, for any ξ > 0, there exists a closed neighbour of implementation of generator G ξ such that E x∼P d,z∼Pz G ξ (z) − x 2 ≤ ξ for all G ξ in the neighbour. Such a neighbour exists because the generator is Lipschitz w.r.t. its parameters. Let the parameter implementation of such neighbour of G ξ be Γ. The Wasserstein distance between G ξ and P d is ξ. Since the function f and the discriminator are L f -Lipschitz and L-Lipschitz, respectively, we have DISPLAYFORM0. Thus, for any fixed γ, we have DISPLAYFORM1 which implies that sup θ∈R t Φ(γ 1, ..., γ I ; θ) = 2f (1/2) for all γ 1,..., γ I ∈ Γ. So we have V = 2f (1/2). This means that the discriminator cannot do much better than a random guess. The above analysis implies that the equilibrium is achieved when D θ * only outputs 1/2. Denote by Θ the small closed neighbour of such θ * such that Φ(γ 1, ..., γ I ; θ) is concave w.r.t. θ ∈ Θ for any fixed γ 1,..., γ I ∈ Γ. We thus focus on the loss on Θ ⊆ R t and Γ ⊆ R g: DISPLAYFORM2 Since Φ(γ 1, ..., γ I ; θ) is concave w.r.t. θ ∈ Θ for all γ 1,..., γ I ∈ Γ, by Corollary 2, we have DISPLAYFORM3 The optimal implementations of γ 1,..., γ I is achieved by argmin γ1,...,γ I ∈Γ sup θ∈Θ DISPLAYFORM4 Proof. We define DISPLAYFORM5 Clearly, the vanilla GAN optimization can be understood as projecting under L: DISPLAYFORM6 In the Stackelberg GAN setting, we are projecting under a different distanceL which is defined as DISPLAYFORM7 We note that f is strictly concave and the discriminator has capacity large enough implies the followings: L(P 1, P 2), as a function of P 2, achieves the global minimum if and only if P 2 = P 1. The theorem then follows from this fact and.E NETWORK SETUP Adam(β 1 = 0.5, β 2 = 0.999) Weight, bias initialization N (µ = 0, σ = 0.01), 0

We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design, with theoretical guarantees.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.