Question: represent the text to find a very short summary (1-2 sentences max)

Question: Reinforcement learning (RL) with value-based methods (e.g., Q-learning) has shown success in a variety of domains such as games and recommender systems (RSs). When the action space is finite, these algorithms implicitly finds a policy by learning the optimal value function, which are often very efficient. However, one major challenge of extending Q-learning to tackle continuous-action RL problems is that obtaining optimal Bellman backup requires solving a continuous action-maximization (max-Q) problem. While it is common to restrict the parameterization of the Q-function to be concave in actions to simplify the max-Q problem, such a restriction might lead to performance degradation. Alternatively, when the Q-function is parameterized with a generic feed-forward neural network (NN), the max-Q problem can be NP-hard. In this work, we propose the CAQL method which minimizes the Bellman residual using Q-learning with one of several plug-and-play action optimizers. In particular, leveraging the strides of optimization theories in deep NN, we show that max-Q problem can be solved optimally with mixed-integer programming (MIP)---when the Q-function has sufficient representation power, this MIP-based optimization induces better policies and is more robust than counterparts, e.g., CEM or GA, that approximate the max-Q solution. To speed up training of CAQL, we develop three techniques, namely (i) dynamic tolerance, (ii) dual filtering, and (iii) clustering. To speed up inference of CAQL, we introduce the action function that concurrently learns the optimal policy. To demonstrate the efficiency of CAQL we compare it with state-of-the-art RL algorithms on benchmark continuous control problems that have different degrees of action constraints and show that CAQL significantly outperforms policy-based methods in heavily constrained environments. Reinforcement learning (RL) has shown success in a variety of domains such as games and recommender systems (RSs) . When the action space is finite, valuebased algorithms such as Q-learning , which implicitly finds a policy by learning the optimal value function, are often very efficient because action optimization can be done by exhaustive enumeration. By contrast, in problems with a continuous action spaces (e.g., robotics ), policy-based algorithms, such as policy gradient (PG) or cross-entropy policy search (CEPS) , which directly learn a return-maximizing policy, have proven more practical. Recently, methods such as ensemble critic and entropy regularization have been developed to improve the performance of policy-based RL algorithms. Policy-based approaches require a reasonable choice of policy parameterization. In some continuous control problems, Gaussian distributions over actions conditioned on some state representation is used. However, in applications such as RSs, where actions often take the form of high-dimensional item-feature vectors, policies cannot typically be modeled by common action distributions. Furthermore, the admissible action set in RL is constrained in practice, for example, when actions must lie within a specific range for safety . In RSs, the admissible actions are often random functions of the state . In such cases, it is non-trivial to define policy parameterizations that handle such factors. On the other hand, value-based algorithms are wellsuited to these settings, providing potential advantage over policy methods. Moreover, at least with linear function approximation , under reasonable assumptions, Q-learning converges to optimality, while such optimality guarantees for non-convex policy-based methods are generally limited . Empirical also suggest that value-based methods are more data-efficient and less sensitive to hyper-parameters . Of course, with large action spaces, exhaustive action enumeration in value-based algorithms can be expensive--one solution is to represent actions with continuous features . The main challenge in applying value-based algorithms to continuous-action domains is selecting optimal actions (both at training and inference time). Previous work in this direction falls into three broad categories. The first solves the inner maximization of the (optimal) Bellman residual loss using global nonlinear optimizers, such as the cross-entropy method (CEM) for QT-Opt , gradient ascent (GA) for actor-expert , and action discretization (; ;). However, these approaches do not guarantee optimality. The second approach restricts the Q-function parameterization so that the optimization problem is tractable. For instance, wire-fitting approximates Q-values piecewise-linearly over a discrete set of points, chosen to ensure the maximum action is one of the extreme points. The normalized advantage function (NAF) constructs the state-action advantage function to be quadratic, hence analytically solvable. Parameterizing the Q-function with an input-convex neural network ensures it is concave. These restricted functional forms, however, may degrade performance if the domain does not conform to the imposed structure. The third category replaces optimal Q-values with a "soft" counterpart : an entropy regularizer ensures that both the optimal Q-function and policy have closed-form solutions. However, the sub-optimality gap of this soft policy scales with the interval and dimensionality of the action space . Motivated by the shortcomings of prior approaches, we propose Continuous Action Q-learning (CAQL), a Q-learning framework for continuous actions in which the Q-function is modeled by a generic feed-forward neural network. 1 Our contribution is three-fold. First, we develop the CAQL framework, which minimizes the Bellman residual in Q-learning using one of several "plug-andplay" action optimizers. We show that "max-Q" optimization, when the Q-function is approximated by a deep ReLU network, can be formulated as a mixed-integer program (MIP) that solves max-Q optimally. When the Q-function has sufficient representation power, MIP-based optimization induces better policies and is more robust than methods (e.g., CEM, GA) that approximate the max-Q solution. Second, to improve CAQL's practicality for larger-scale applications, we develop three speed-up techniques for computing max-Q values: (i) dynamic tolerance; (ii) dual filtering; and (iii) clustering. Third, we compare CAQL with several state-of-the-art RL algorithms on several benchmark problems with varying degrees of action constraints. Value-based CAQL is generally competitive, and outperforms policy-based methods in heavily constrained environments, sometimes significantly. We also study the effects of our speed-ups through ablation analysis. We consider an infinite-horizon, discounted Markov decision process with states X, (continuous) action space A, reward function R, transition kernel P, initial state distribution β and discount factor γ ∈, all having the usual meaning. A (stationary, Markovian) policy π specifies a distribution π(·|x) over actions to be taken at state x. Let ∆ be the set of such policies. The expected cumulative return of π ∈ ∆ is J(π):= E[∞ t=0 γ t r t | P, R, x 0 ∼ β, π]. An optimal policy π * satisfies π * ∈ arg max π∈∆ J(π). The Bellman operator F [Q](x, a) = R(x, a) + γ x ∈X P (x |x, a) max a ∈A Q(x, a) over state-action value function Q has unique fixed point Q * (x, a) , which is the optimal Q-function Q * (x, a) = E [∞ t=0 γ t R(x t, a t) | x 0 = x, a 0 = a, π * ]. An optimal (deterministic) policy π * can be extracted from Q *: π * (a|x) = 1{a = a * (x)}, where a * (x) ∈ arg max a Q * (x, a). For large or continuous state/action spaces, the optimal Q-function can be approximated, e.g., using a deep neural network (DNN) as in DQN . In DQN, the value function Q θ is updated using the value label r + γ max a Q θ target (x, a), where Q θ target is a target Q-function. Instead of training these weights jointly, θ target is updated in a separate iterative fashion using the previous θ for a fixed number of training steps, or by averaging θ target ← τ θ + (1 − τ)θ target for some small momentum weight τ ∈ . DQN is off-policy-the target is valid no matter how the experience was generated (as long as it is sufficiently exploratory). Typically, the loss is minimized over mini-batches B of past data (x, a, r, x) sampled from a large experience replay buffer R . One common loss function for training Q θ * is mean squared Bellman error: 2. Under this loss, RL can be viewed as 2 -regression of Q θ (·, ·) w.r.t. target labels r + γ max a Q θ target (x, a). We augment DQN, using double Q-learning for more stable training , whose loss is: A hinge loss can also be used in Q-learning, and has connections to the linear programming (LP) formulation of the MDP . The optimal Q-network weights can be specified as: To stabilize training, we replace the Q-network of the inner maximization with the target Q-network and the optimal Q-value with the double-Q label, giving (see Appendix A for details): In this work, we assume the Q-function approximation Q θ to be a feed-forward network. Specifically, let Q θ be a K-layer feed-forward NN with state-action input (x, a) (where a lies in a ddimensional real vector space) and hidden layers arranged according to the equations: where (W j, b j) are the multiplicative and bias weights, c is the output weight of the Q-network, are the weights of the Q-network,ẑ j denotes pre-activation values at layer j, and h(·) is the (component-wise) activation function. For simplicity, in the following analysis, we restrict our attention to the case when the activation functions are ReLU's. We also assume that the action space A is a d-dimensional ∞ -ball B ∞ (a, ∆) with some radius ∆ > 0 and center a. Therefore, at any arbitrary state x ∈ X the max-Q problem can be re-written as q * While the above formulation is intuitive, the nonlinear equality constraints in the neural network formulation makes this problem non-convex and NP-hard . Policy-based methods (; ;) have been widely-used to handle continuous actions in RL. However, they suffer from several well-known difficulties, e.g., (i) modeling high-dimensional action distributions, (ii) handling action constraints, and (iii) data-inefficiency. Motivated by earlier work on value-based RL methods, such as QTOpt and actor-expert , we propose Continuous Action Q-learning (CAQL), a general framework for continuous-action value-based RL, in which the Qfunction is parameterized by a NN (Eq. 3). One novelty of CAQL is the formulation of the "max-Q" problem, i.e., the inner maximization in and, as a mixed-integer programming (MIP). The benefit of the MIP formulation is that it guarantees that we find the optimal action (and its true bootstrapped Q-value) when computing target labels (and at inference time). We show empirically that this can induce better performance, especially when the Q-network has sufficient representation power. Moreover, since MIP can readily model linear and combinatorial constraints, it offers considerable flexibility when incorporating complex action constraints in RL. That said, finding the optimal Q-label (e.g., with MIP) is computationally intensive. To alleviate this, we develop several approximation methods to systematically reduce the computational demands of the inner maximization. In Sec. 3.2, we introduce the action function to approximate the arg max-policy at inference time, and in Sec. 4 we propose three techniques, dynamic tolerance, dual filtering, and clustering, to speed up max-Q computation during training. In this section, we illustrate how the max-Q problem, with the Q-function represented by a ReLU network, can be formulated as a MIP, which can be solved using off-the-shelf optimization packages (e.g., SCIP , CPLEX , Gurobi ). In addition, we detail how approximate optimizers, specifically, gradient ascent (GA) and the cross-entropy method (CEM), can trade optimality for speed in max-Q computation within CAQL. A trained feed-forward ReLU network can be modeled as a MIP by formulating the nonlinear activation function at each neuron with binary constraints. Specifically, for a ReLU with pre-activation function of form z = max{0, w x + b}, where and, u ∈ R d are the weights, bias and lower-upper bounds respectively, consider the following set with a binary variable ζ indicating whether the ReLU is active or not:. In this formulation, both M + = max x∈[,u] w x + b and M − = min x∈[,u] w x + b can be computed in linear time in d. We assume M + > 0 and M − < 0, otherwise the function can be replaced by z = 0 or z = w x + b. These constraints ensure that z is the output of the ReLU: If ζ = 0, then they are reduced to z = 0 ≥ w x+b, and if ζ = 1, then they become z = w x+b ≥ 0. This can be extended to the ReLU network in by chaining copies of intermediate ReLU formulations. More precisely, if the ReLU Q-network has m j neurons in layer j ∈ {2, . . ., K}, for any given state x ∈ X, the max-Q problem can be reformulated as the following MIP:,..., K}, i ∈ {1, . . ., m j}, where 1 = a − ∆, u 1 = a + ∆ are the (action) input-bound vectors. Since the output layer of the ReLU NN is linear, the MIP objective is linear as well. Here, W j,i ∈ R mj and b j,i ∈ R are the weights and bias of neuron i in layer j. Furthermore, j, u j are interval bounds for the outputs of the neurons in layer j for j ≥ 2, and computing them can be done via interval arithmetic or other propagation methods from the initial action space bounds (see Appendix C for details). As detailed by , this can be further tightened with additional constraints, and its implementation can be found in the tf.opt package described therein. As long as these bounds are redundant, having these additional box constraints will not affect optimality. We emphasize that the MIP returns provably global optima, unlike GA and CEM. Even when interrupted with stopping conditions such as a time limit, MIP often produces high-quality solutions in practice. In theory, this MIP formulation can be solved in time exponential on the number of ReLUs and polynomial on the input size (e.g., by naively solving an LP for each binary variable assignment). In practice however, a modern MIP solver combines many different techniques to significantly speed up this process, such as branch-and-bound, cutting planes, preprocessing techniques, and primal heuristics . Versions of this MIP model have been used in neural network verification (; ; ; ; ; ;) and analysis , but its application to RL is novel. also proposed a MIP formulation to solve the planning problem with non-linear state transition dynamics model learned with a NN, it is different than ours, which solves the max-Q problem. Gradient Ascent GA is a simple first-order optimization method for finding the (local) optimum of a differentiable objective function, such as a neural network Qfunction. At any state x ∈ X, given a "seed" action a 0, the optimal action arg max a Q θ (x, a) is computed iteratively by a t+1 ← a t + η∇ a Q θ (x, a), where η > 0 is a step size (either a tunable parameter or computed using back-tracking line search ). This process repeats until convergence, |Q θ (x, a t+1) − Q θ (x, a t)| <, or a maximum iteration count is reached. Cross-Entropy Method CEM is a derivative-free optimization algorithm. At any given state x ∈ X, it samples a batch of N actions {a i} N i=1 from A using a fixed distribution (e.g., a Gaussian) and ranks the corresponding Q-values. Using the top K < N actions, it then updates the sampling distribution, e.g., using the sample mean and covariance to update the Gaussian. This is repeated until convergence or a maximum iteration count is reached. In traditional Q-learning, the policy π * is "implemented" by acting greedily w.r.t. the learned Qfunction: π * (x) = arg max a Q θ (x, a). 3 However, computing the optimal action can be expensive in the continuous case, which may be especially problematic at inference time (e.g., when computational power is limited in, say embedded systems, or real-time response is critical). To mitigate the problem, we can use an action function π w: X → A-effectively a trainable actor network-to approximate the greedy-action mapping π *. We train π w using training data B = {(, where q * i is the max-Q label at state x i . Action function learning is then simply a supervised regression problem: 2 . This is similar to the notion of "distilling" an optimal policy from max-Q labels, as in actor-expert . Unlike actorexpert-a separate stochastic policy network is jointly learned with the Q-function to maximize the likelihood with the underlying optimal policy-our method learns a state-action mapping to approximate arg max a Q θ (x, a)-this does not require distribution matching and is generally more stable. The use of action function in CAQL is simply optional to accelerate data collection and inference. In this section, we propose three methods to speed up the computationally-expensive max-Q solution during training: (i) dynamic tolerance, (ii) dual filtering, and (iii) clustering. Dynamic Tolerance Tolerance plays a critical role in the stopping condition of nonlinear optimizers. Intuitively, in the early phase of CAQL, when the Q-function estimate has high Bellman error, it may be wasteful to compute a highly accurate max-Q label when a crude estimate can already guide the gradient of CAQL to minimize the Bellman residual. We can speed up the max-Q solver by dynamically adjusting its tolerance τ > 0 based on (a) the TD-error, which measures the estimation error of the optimal Q-function, and (b) the training step t > 0, which ensures the bias of the gradient (induced by the sub-optimality of max-Q solver) vanishes asymptotically so that CAQL converges to a stationary point. While relating tolerance with the Bellman residual is intuitive, it is impossible to calculate that without knowing the max-Q label. To resolve this circular dependency, notice that the action function π w approximates the optimal policy, i.e., π w (·|x) ≈ arg max a Q θ (x, ·). We therefore replace the optimal policy with the action function in Bellman residual and propose the dynamic tolerance:, where k 1 > 0 and k 2 ∈ are tunable parameters. Under standard assumptions, CAQL with dynamic tolerance {τ t} converges a.s. to a stationary point (Thm. 1, ). The main motivation of dual filtering is to reduce the number of max-Q problems at each CAQL training step. For illustration, consider the formulation of hinge Q-learning in. Denote by q * x,θ target the max-Q label w.r.t. the target Q-network and next state x. The structure of the hinge penalty means the TD-error corresponding to sample (x, a, x, r) is inactive whenever q * x,θ target ≤ (Q θ (x, a) − r)/γ-this data can be discarded. In dual filtering, we efficiently estimate an upper bound on q * x,θ target using some convex relaxation to determine which data can be discarded before max-Q optimization. Specifically, recall that the main source of non-convexity in comes from the equality constraint of the ReLU activation function at each NN layer. Similar to MIP formulation, assume we have component-wise bounds We use this approximation to define the relaxed NN equations, which replace the nonlinear equality constraints in with the convex set H(l, u). We denote the optimal Q-value w.r.t. the relaxed NN asq * x, which is by definition an upper bound on q * x i. Hence, the condition:q * x,θ target ≤ (Q θ (x, a) − r)/γ is a conservative certificate for checking whether the data (x, a, x, r) is inactive. For further speed up, we estimateq * x with its dual upper bound (see Appendix C for derivations)q, where ν is defined by the following recursion "dual" network: ) · 1{s ∈ I j}, and replace the above certificate with an even more conservative one: Although dual filtering is derived for hinge Q-learning, it also applies to the 2 -loss counterpart by replacing the optimal value q One can utilize the inactive samples in the π w -learning problem by replacing the max-Q label q * x,θ with its dual approximation q x,θ. Since q x,θ ≥ q * x,θ, this replacement will not affect optimality. Clustering To reduce the number of max-Q solves further still, we apply online state aggregation , which picks a number of centroids from the batch of next states B as the centers of p-metric balls with radius b > 0, such that the union of these balls form a minimum covering of B. Specifically, at training step t ∈ {0, 1, . . .}, denote by C t (b) ⊆ B the set of next-state centroids. For each next state c ∈ C t (b), we compute the max-Q value q * c,θ target = max a Q θ target (c, a), where a * c is the corresponding optimal action. For all remaining next states x ∈ B \ C t (b), we approximate their max-Q values via first-order Taylor series expansionq x,θ target:= q centroid to x, i.e., c ∈ arg min c ∈Ct(b) x −c p. By the envelope theorem for arbitrary choice sets In this approach the cluster radius r > 0 controls the number of max-Q computations, which trades complexity for accuracy in Bellman residual estimation. This parameter can either be a tuned or adjusted dynamically (similar to dynamic tolerance), e.g., r t = k 3 · k t 4 with hyperparameters k 3 > 0 and k 4 ∈. Analogously, with this exponentially-decaying cluster radius schedule we can argue that the bias of CAQL gradient (induced by max-Q estimation error due to clustering) vanishes asymptotically, and the corresponding Q-function converges to a stationary point. To combine clustering with dual filtering, we define B df as the batch of next states that are inconclusive after dual filtering, i.e., B df = {x ∈ B : q x,θ target > (Q θ (x, a) − r)/γ}. Then instead of applying clustering to B we apply this method onto the refined batch B df. To illustrate the effectiveness of CAQL, we (i) compare several CAQL variants with several state-ofthe-art RL methods on multiple domains, and (ii) assess the trade-off between max-Q computation speed and policy quality via ablation analysis. Comparison with Baseline RL Algorithms We compare CAQL with three baseline methods, DDPG and TD3 -two popular policy-based deep RL algorithms-and NAF , a value-based method using an action-quadratic Q-function. We train CAQL using three different max-Q optimizers, MIP, GA, and CEM. Note that CAQL-CEM counterpart is similar to QT-Opt and CAQL-GA reflects some aspects actor-expert . These CAQL variants allow assessment of the degree to which policy quality is impacted by Q-learning with optimal Bellman residual (using MIP) rather than an approximation (using GA or CEM), at the cost of steeper computation. To match the implementations of the baselines, we use 2 loss when training CAQL. Further ablation analysis on CAQL with 2 loss vs. hinge loss is provided in Appendix E. We evaluate CAQL on one classical control benchmark (Pendulum) and five MuJoCo benchmarks (Hopper, Walker2D, HalfCheetah, Ant, Humanoid). Different than most previous work, we evaluate the RL algorithms on domains not just with default action ranges, but also using smaller, constrained action ranges (see Table 6 in Appendix D for action ranges used in our experiments). 4 The motivation for this is two-fold: (i) To simulate real-world problems , where the restricted ranges represent the safe/constrained action sets; (ii) To validate the hypothesis that action-distribution learning in policy-based methods cannot easily handle such constraints, while CAQL does so, illustrating its flexibility. We reduce episode limits from 1000 to 200 steps and use small networks to accommodate the MIP. Both changes lead to lower returns than that reported in state-of-the-art RL benchmarks . Details on network architectures and hyperparameters are described in Appendix D. Policy performance is evaluated every 1000 training iterations, using a policy with no exploration. Each measurement is an average return over 10 episodes, each generated using a separate random seed. To smooth learning curves, data points are averaged over a sliding window of size 3. Similar to the setting of , CAQL measurements are based on trajectories that are generated by the learned action function instead of the optimal action w.r.t. the Q-function. Table 1 shows the average return of CAQL and the baselines under the best hyperparameter configurations. CAQL significantly outperforms NAF on most benchmarks, as well as DDPG and TD3 on 10 of 14 benchmarks. Of all the CAQL policies, those trained using MIP are among the best performers in all the benchmarks except Ant [-0.25, 0 .25] and Humanoid [-0.25, 0.25]. This verifies our conjecture about CAQL: Q-learning with optimal Bellman residual (using MIP) performs better than using approximation (using GA, CEM) when the Q-function has sufficient representation power (which is more likely in low-dimensional tasks). Moreover, CAQL-MIP policies have slightly lower variance than those trained with GA and CEM on most benchmarks. Table 2 shows summary statistics of the returns of CAQL and the baselines on all 320 configurations (32 hyperparameter combinations × 10 random seeds) and illustrates the sensitivity to hyperparameters of each method. CAQL is least sensitive in 13 of 14 tasks, and policies trained using MIP optimization, specifically, are best in 8 of 14 tasks. This corroborates the hypothesis that value- Table 2: The mean ± standard deviation of (95-percentile) final returns over all 320 configurations (32 hyper parameter combinations×10 random seeds). The full training curves are given in Figure 4 in Appendix E. CAQL-MIP policies are least sensitive to hyper parameters on 8/14 benchmarks. based methods are generally more robust to hyperparameters than their policy-based counterparts. Table 9 in Appendix E.1 compares the speed (in terms of average elapsed time) of various max-Q solvers (MIP, GA, and CEM), with MIP clearly the most computationally intensive. We note that CAQL-MIP suffers from performance degradation in several high-dimensional environments with large action ranges (e.g., Ant [-0.25, 0 .25] and Humanoid [-0.25, 0.25] ). In these experiments, its performance is even worse than that of CAQL-GA or CAQL-CEM. We speculate that this is due to the fact that the small ReLU NN (32 × 16) doesn't have enough representation power to accurately model the Q-functions in more complex tasks, and therefore optimizing for the true max-Q value using an inaccurate function approximation impedes learning. Ablation Analysis We now study the effects of using dynamic tolerance, dual filtering, and clustering on CAQL via two ablation analyses. For simplicity, we experiment on standard benchmarks (with full action ranges), and primarily test CAQL-GA using an 2 loss. Default values on tolerance and maximum iteration are 1e-6 and 200, respectively. Table 3 shows how reducing the number of max-Q problems using dual filtering and clustering affects performance of CAQL. Dual filtering (DF) manages to reduce the number of max-Q problems (from 3.2% to 26.5% across different benchmarks), while maintaining similar performance with the unfiltered CAQL-GA. On top of dual filtering we apply clustering (C) to the set of inconclusive next states B df, in which the degree of approximation is controlled by the cluster radius. With a small cluster radius (e.g., b = 0.1), clustering further reduces max-Q solves without significantly impacting training performance (and in some cases it actually improves performance), though further increasing the radius would significant degrade performance. To illustrate the full trade-off of max-Q reduction versus policy quality, we also include the Dual method, which eliminates all max-Q computation with the dual approximation. Table 4 shows how dynamic tolerance influences the quality of CAQL policies. Compared with the standard algorithm, with a large tolerance (τ = 100) GA achieves a notable speed up (with only 1 step per max-Q optimization) in training but incurs a loss in performance. GA with dynamic tolerance atttains the best of both worlds-it significantly Table 3: Ablation analysis on CAQL-GA with dual filtering and clustering, where both the mean ± standard deviation of (95-percentile) final returns and the average %-max-Q-reduction (in parenthesis) are based on the best configuration. See Figure 5 in Appendix E for training curves. Table 4: Ablation analysis on CAQL-GA with dynamic tolerance, where both the mean ± standard deviation of (95-percentile) final returns and the average number of GA iterations (in parenthesis) are based on the best configuration. See Figure 7 in Appendix E for training curves. NOTE: In (*) the performance significantly drops after hitting the peak, and learning curve does not converge. reduces inner-maximization steps (from 29.5% to 77.3% across different problems and initial τ settings), while achieving good performance. Additionally, Table 5 shows the of CAQL-MIP with dynamic tolerance (i.e., optimality gap). This method significantly reduces both median and variance of the MIP elapsed time, while having better performance. Dynamic tolerance eliminates the high latency in MIP observed in the early phase of training (see Figure 1 and 2). Table 5: Ablation analysis on CAQL-MIP with dynamic tolerance, where both the mean ± standard deviation of (95-percentile) final returns and the (median, standard deviation) of the elapsed time κ (in msec) are based on the best configuration. See Figure 11 in Appendix E for training curves. We proposed Continuous Action Q-learning (CAQL), a general framework for handling continuous actions in value-based RL, in which the Q-function is parameterized by a neural network. While generic nonlinear optimizers can be naturally integrated with CAQL, we illustrated how the inner maximization of Q-learning can be formulated as mixed-integer programming when the Qfunction is parameterized with a ReLU network. CAQL (with action function learning) is a general Q-learning framework that includes many existing value-based methods such as QT-Opt and actorexpert. Using several benchmarks with varying degrees of action constraint, we showed that the policy learned by CAQL-MIP generally outperforms those learned by CAQL-GA and CAQL-CEM; and CAQL is competitive with several state-of-the-art policy-based RL algorithms, and often outperforms them (and is more robust) in heavily-constrained environments. Future work includes: extending CAQL to the full batch learning setting, in which the optimal Q-function is trained using only offline data; speeding up the MIP computation of the max-Q problem to make CAQL more scalable; and applying CAQL to real-world RL problems. Consider an MDP with states X, actions A, transition probability function P, discount factor γ ∈, reward function R, and initial state distribution β. We want to find an optimal Q-function by solving the following optimization problem: The formulation is based on the LP formulation of MDP (see for more details). Here the distribution p(x, a) is given by the data-generating distribution of the replay buffer B. (We assume that the replay buffer is large enough such that it consists of experience from almost all state-action pairs.) It is well-known that one can transform the above constrained optimization problem into an unconstrained one by applying a penalty-based approach (to the constraints). For simplicity, here we stick with a single constant penalty parameter λ ≥ 0 (instead of going for a state-action Lagrange multiplier and maximizing that), and a hinge penalty function (·) +. With a given penalty hyper-parameter λ ≥ 0 (that can be separately optimized), we propose finding the optimal Q-function by solving the following optimization problem: Furthermore, recall that in many off-policy and offline RL algorithms (such as DQN), samples in form of are independently drawn from the replay buffer, and instead of the optimizing the original objective function, one goes for its unbiased sample average approximation (SAA). However, viewing from the objective function of problem, finding an unbiased SAA for this problem might be challenging, due to the non-linearity of hinge penalty function (·) +. Therefore, alternatively we turn to study the following unconstrained optimization problem: Using the Jensen's inequality for convex functions, one can see that the objective function in is an upper-bound of that in. Equality of the Jensen's inequality will hold in the case when transition function is deterministic. (This is similar to the argument of PCL algorithm.) Using Jensen's inequality one justifies that optimization problem is indeed an eligible upper-bound optimization to problem. Recall that p(x, a) is the data-generation distribution of the replay buffer B. The unbiased SAA of problem is therefore given by where are the N samples drawn independently from the replay buffer. In the following, we will find the optimal Q function by solving this SAA problem. In general when the state and action spaces are large/uncountable, instead of solving the Q-function exactly (as in the tabular case), we turn to approximate the Q-function with its parametrized form Q θ, and optimize the set of real weights θ (instead of Q) in problem. Sample an initial state x 0 from the initial distribution 8: Select action a = clip(π w (x) + N (0, σ), l, u) 10: Execute action a and observe reward r and new state x +1 Store transition (x, a, r, x +1) in Replay Buffer R for s ← 1,..., S do CAQL Training; S = 20 by default 13: Sample a random minibatch B of |B| transitions {( Initialize the refined batches B df ← B and B c ← B; For each (x i, a i, r i, x i) ∈ B df ∩ B c, compute optimal action a i using OPT(DTol): and the corresponding TD targets: For each (x i, a i, r i, x i) ∈ B \ (B c ∩ B df), compute the approximate TD target: Update the Q-function parameters: Update the action function parameters: Update the target Q-function parameters: Decay the Gaussian noise: Recall that the Q-function NN has a nonlinear activation function, which can be viewed as a nonlinear equality constraint, according to the formulation in. To tackle this constraint, proposed a convex relaxation of the ReLU non-linearity. Specifically, first, they assume that for given x ∈ X and a ∈ B ∞ (a) such that z 1 = (x, a), there exists a collection of component-wise bounds (l j, u j), j = 2,..., K − 1 such that l j ≤ẑ j ≤ u j. As long as the bounds are redundant, adding these constraints into primal problem q * x does not affect the optimal value. Second, the ReLU non-linear equality constraint is relaxed using a convex outer-approximation. In particular, for a scalar input a within the real interval [l, u], the exact ReLU non-linearity acting on a is captured by the set Its convex outer-approximation is given by: Analogously to, define the relaxed NN equations as: where the third equation above is understood to be component-wise across layer j for each j ∈ {2, . . ., K − 1}, i.e., where n j is the dimension of hidden layer j. Using the relaxed NN equations, we now propose the following relaxed (convex) verification problem: where δ Λ (·) is the indicator function for set Λ (i.e., δ Λ (x) = 0 if x ∈ Λ and ∞ otherwise). Note that the indicator for the vector-ReLU in cost function above is understood to be component-wise, i.e., The optimal value of the relaxed problem, i.e.,q * x is an upper bound on the optimal value for original problem q * x i. Thus, the certification one can obtain is the following: ifq * x ≤ (Q θ (x, a) − r)/γ, then the sample (x, a, x) is discarded for inner maximization. However, ifq * x > (Q θ (x, a) − r)/γ, the sample (x, a, x) may or may not have any contribution to the TD-error in the hinge loss function. To further speed up the computation of the verification problem, by looking into the dual variables of problem, in the next section we propose a numerically efficient technique to estimate a suboptimal, upper-bound estimate toq * x, namely q x. Therefore, one verification criterion on whether a sample drawn from replay buffer should be discarded for inner-maximization is check whether the following inequality holds: C.1 SUB-OPTIMAL SOLUTION TO THE RELAXED PROBLEM In this section, we detail the sub-optimal lower bound solution to the relaxed problem in as proposed in. Let ν j, j = 2,..., K denote the dual variables for the linear equality constraints in problem. The Lagrangian for the relaxed problem in is given by: Defineν j: = W j ν j+1 for j = 1,..., K − 1, and defineν Recall that for a real vector space X ⊆ R n, let X * denote the dual space of X with the standard pairing ·, ·: X × X * → R. For a real-valued function f: X → R ∪ {∞, −∞}, let f *: X * → R ∪ {∞, −∞} be its convex conjugate, defined as: f * (y) = − inf x∈X (f (x) − y, x ) = sup x∈X (y, x − f (x)), ∀y ∈ X *. Therefore, the conjugate for the vector-ReLU indicator above takes the following component-wise structure: Now, the convex conjugate of the set indicator function is given by the set support function. Thus, where · q is the l p -dual norm defined by the identity 1/p + 1/q = 1. To compute the convex conjugate for the ReLU relaxation, we analyze the scalar definition as provided in. Specifically, we characterize δ H (l,u) (p, q) defined by the scalar bounds (l, u), for the dual vector (p, q) ∈ R 2. There exist 3 possible cases: Case I: l < u ≤ 0: Case II: 0 ≤ l < u: Case III: l < 0 < u: For this case, the sup will occur either on the line −ux + (u − l)y = −ul or at the origin. Thus, Applying these in context of equation, we calculate the Lagrange multipliers by considering the following cases. Case I: l j (s) < u j (s) ≤ 0: In this case, since z j (s) = 0 regardless of the value ofẑ j (s), one can simply remove these variables from problems and by eliminating the j th row of W j−1 and b j−1 and the j th column of W i. Equivalently, from, one can remove their contribution in by setting ν j (s) = 0. Case II: 0 ≤ l j (s) < u j (s): In this case, the ReLU non-linearity for (ẑ j (s), z j (s)) in problems and may be replaced with the convex linear equality constraint z j (s) =ẑ j (s) with associated dual variable µ. Within the Lagrangian, this would in a modification of the term Minimizing this over (ẑ j (s), z j (s) ), a non-trivial lower bound (i.e., 0) is obtained only if ν j (s) = ν j (s) = µ. Equivalently, from, we set ν j (s) =ν j (s). Case III: For the non-trivial third case, where l j (s) < 0 < u j (s), notice that due toν, the dual function g(ν) is not decoupled across the layers. In order to get a sub-optimal, but analytical solution to the dual optimization, we will optimize each term within the first sum in independently. To do this, notice that the quantity in sub-case I in is strictly greater than the other two sub-cases. Thus, the best bound is obtained by using the third sub-case, which corresponds to setting:. Combining all the previous analysis, we now calculate the dual of the solution to problem. Let I Using the above case studies, a sub-optimal (upper-bound) dual solution to the primal solutionJ x in problem is given by where ν is defined by the following recursion, termed the "dual" network: and D j is a diagonal matrix with C.2 COMPUTING PRE-ACTIVATION BOUNDS For k ∈ {3, . . ., K − 1}, define the k−partial NN as the set of equations: Finding the lower bound l k forẑ k involves solving the following problem: where e s is a one-hot vector with the non-zero element in the s-th entry, for s ∈ {1, . . ., n k}. Similarly, we obtain u k by maximizing the objective above. Assuming we are given bounds {l j, u j} k−1 j=2, we can employ the same convex relaxation technique and approximate dual solution as for the verification problem (since we are simply optimizing a linear function of the output of the first k layers of the NN). Doing this recursively allows us to compute the bounds {l j, u j} for j = 3,..., K − 1. The recursion is given in Algorithm 1 in and is based on the matrix form of the recursion in, i.e., with c replaced with I and −I, so that the quantity in We use a two hidden layer neural network with ReLU activation (32 units in the first layer and 16 units in the second layer) for both the Q-function and the action function. The input layer for the Q-function is a concatenated vector of state representation and action variables. The Q-function has a single output unit (without ReLU). The input layer for the action function is only the state representation. The output layer for the action function has d units (without ReLU), where d is the action dimension of a benchmark environment. We use SCIP 6.0.0 for the MIP solver. A time limit of 60 seconds and a optimality gap limit of 10 −4 are used for all experiments. For GA and CEM, a maximum iterations of 20 and a convergence threshold of 10 are used for all experiments. E ADDITIONAL EXPERIMENTAL E.1 OPTIMIZER SCALABILITY Table 9 shows the average elapsed time of various optimizers computing max-Q in the experiment setup described in Appendix D. MIP is more robust to action dimensions than GA and CEM. MIP latency depends on the state of neural network weights. It takes longer time with highly dense NN weights, but on the other hand, it can be substantially quicker with sparse NN weights. Figure 1 shows the average elapsed time of MIP over training steps for various benchmarks. We have observed that MIP is very slow in the beginning of the training phase but it quickly becomes faster. This trend is observed for most benchmarks except Humanoid. We speculate that the NN weights for the Q-function are dense in the beginning of the training phase, but it is gradually structurized (e.g, sparser weights) so that it becomes an easier problem for MIP. Table 9: The (median, standard deviation) for the average elapsed time κ (in msec) of various solvers computing max-Q problem. (n) Humanoid [-0.25,0.25] Figure 4: The mean return over all 320 configurations (32 hyper parameter combinations × 10 random seeds). Shaded area is ± standard deviation. Data points are average over a sliding window of size 3. The length of an episode is limited to 200 steps. Table 11: Ablation analysis on CAQL-GA with dynamic tolerance, where both the mean ± standard deviation of (95-percentile) final returns and the average number of GA iterations (in parenthesis) are over all 320 configurations. See Figure 8 in Appendix E for training curves. Table 13: The mean ± standard deviation of (95-percentile) final returns over all 320 configurations (32 hyper parameter combinations × 10 random seeds). The full training curves are given in Figure 10 in Appendix E.

Represent the short summary for finding the corresponding scientific article (long)

A general framework of value-based reinforcement learning for continuous control

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative Adversarial Networks (GANs) have proven to be a powerful framework for learning to draw samples from complex distributions. However, GANs are also notoriously difficult to train, with mode collapse and oscillations a common problem. We hypothesize that this is at least in part due to the evolution of the generator distribution and the catastrophic forgetting tendency of neural networks, which leads to the discriminator losing the ability to remember synthesized samples from previous instantiations of the generator. Recognizing this, our contributions are twofold. First, we show that GAN training makes for a more interesting and realistic benchmark for continual learning methods evaluation than some of the more canonical datasets. Second, we propose leveraging continual learning techniques to augment the discriminator, preserving its ability to recognize previous generator samples. We show that the ing methods add only a light amount of computation, involve minimal changes to the model, and in better overall performance on the examined image and text generation tasks. Generative Adversarial Networks BID6 (GANs) are a popular framework for modeling draws from complex distributions, demonstrating success in a wide variety of settings, for example image synthesis BID14 and language modeling. In the GAN setup, two agents, the discriminator and the generator (each usually a neural network), are pitted against each other. The generator learns a mapping from an easy-to-sample latent space to a distribution in the data space, which ideally matches the real data's distribution. At the same time, the discriminator aims to distinguish the generator's synthesized samples from the real data samples. When trained successfully, GANs yield impressive ; in the image domain for example, synthesized images from GAN models are significantly sharper and more realistic than those of other classes of models BID16. On the other hand, GAN training can be notoriously finicky. One particularly well-known and common failure mode is mode collapse BID0 BID35: instead of producing samples sufficiently representing the true data distribution, the generator maps the entire latent space to a limited subset of the real data space. When mode collapse occurs, the generator does not "converge," in the conventional sense, to a stationary distribution. Rather, because the discriminator can easily learn to recognize a modecollapsed set of samples and the generator is optimized to avoid the discriminator's detection, the two end up playing a never-ending game of cat and mouse: the generator meanders towards regions in the data space the discriminator thinks are real (likely near where the real data lie) while the discriminator chases after it. Interestingly though, if generated samples are plotted through time (as in FIG0), it appears that the generator can revisit previously collapsed modes. At first, this may seem odd. The discriminator was ostensibly trained to recognize that mode in a previous iteration and did so well enough to push the generator away from generating those samples. Why has the discriminator seemingly lost this ability?We conjecture that this oscillation phenomenon is enabled by catastrophic forgetting BID20 BID30: neural networks have a well-known tendency to forget how to complete old tasks while learning new ones. In most GAN models, the discriminator is a binary classifier, with the two classes being the real data and the generator's outputs. Implicit to the training of a standard classifier is the assumption that the data are drawn independently and identically distributed (i.i.d.). Importantly, this assumption does not hold true in GANs: the distribution of the generator class (and thus the discriminator's training data) evolves over time. Moreover, these changes in the generator's distribution are adversarial, designed specifically to deteriorate discriminator performance on the fake class as much as possible. Thus, the alternating training procedure of GANs in actuality corresponds to the discriminator learning tasks sequentially, where each task corresponds to recognizing samples from the generator at that particular point in time. Without any measures to prevent catastrophic forgetting, the discriminator's ability to recognize fake samples from previous iterations will be clobbered by subsequent gradient updates, allowing a mode-collapsed generator to revisit old modes if training runs long enough. Given this tendency, a collapsed generator can wander indefinitely without ever learning the true distribution. With this perspective in mind, we cast training the GAN discriminator as a continual learning problem, leading to two main contributions. (i) While developing systems that learn tasks in a sequential manner without suffering from catastrophic forgetting has become a popular direction of research, current benchmarks have recently come under scrutiny as being unrepresentative to the fundamental challenges of continual learning BID3. We argue that GAN training is a more realistic setting, and one that current methods tend to fail on.(ii) Such a reframing of the GAN problem allows us to leverage relevant methods to better match the dynamics of training the min-max objective. In particular, we build upon the recently proposed elastic weight consolidation BID15 and intelligent synapses BID39. By preserving the discriminator's ability to identify previous generator samples, this memory prevents the generator from simply revisiting past distributions. Adapting the GAN training procedure to account for catastrophic forgetting provides an improvement in GAN performance for little computational cost and without the need to train additional networks. Experiments on CelebA and CIFAR10 image generation and COCO Captions text generation show discriminator continual learning leads to better generations. Consider distribution p real (x), from which we have data samples D real. Seeking a mechanism to draw samples from this distribution, we learn a mapping from an easy-to-sample latent distribution p(z) to a data distribution p gen (x), which we want to match p real (x). This mapping is parameterized as a neural network G φ (z) with parameters φ, termed the generator. The synthesized data are drawn x = G φ (z), with z ∼ p(z). The form of p gen (x) is not explicitly assumed or learned; rather, we learn to draw samples from p gen (x).To provide feedback to G φ (z), we simultaneously learn a binary classifier that aims to distinguish synthesized samples D gen drawn from p gen (x) from the true samples D real. We also parameterize this classifier as a neural network D θ (x) ∈ with parameters θ, with D θ (x) termed the discriminator. By incentivizing the generator to fool the discriminator into thinking its generations are actually from the true data, we hope to learn G φ (z) such that p gen (x) approaches p real (x).These two opposing goals for the generator and discriminator are usually formulated as the following min-max objective: DISPLAYFORM0 At each iteration t, we sample from p gen (x), yielding generated data D gen t. These generated samples, along with samples from D real, are then passed to the discriminator. A gradient descent optimizer nudges θ so that the discriminator takes a step towards maximizing L GAN (θ, φ). Parameters φ are updated similarly, but to minimize L GAN (θ, φ). These updates to θ and φ take place in an alternating fashion. The expectations are approximated using samples from the respective distributions, and therefore learning only requires observed samples D real and samples from p gen (x).The updates to G φ (z) mean that p gen (x) changes as a function of t, perhaps substantially. (x). Because of the catastrophic forgetting effect of neural networks, the ability of D θ (x) to recognize these previous distributions is eventually lost in the pursuit of maximizing L GAN (θ, φ) with respect to only D gen t. This opens the possibility that the generator goes back to generating samples the discriminator had previously learned (and then forgot) to recognize, leading to unstable mode-collapsed oscillations that hamper GAN training (as in FIG0). Recognizing this problem, we propose that the discriminator should be trained with the temporal component of p gen (x) in mind. 3.1 CLASSIC CONTINUAL LEARNING Catastrophic forgetting has long been known to be a problem with neural networks trained on a series of tasks BID20 BID30. While there are many approaches to addressing catastrophic forgetting, here we primarily focus on elastic weight consolidation (EWC) and intelligent synapses (IS). These are meant to illustrate the potential of catastrophic forgetting mitigation to improve GAN learning, with the expectation that this opens up the possibility of other such methods to significantly improve GAN training, at low additional computational cost. To derive the EWC loss, BID15 frames training a model as finding the most probable values of the parameters θ given the data D. For two tasks, the data are assumed partitioned into independent sets according to the task, and the posterior for Task 1 is approximated as a Gaussian with mean centered on the optimal parameters for Task 1 θ * 1 and diagonal precision given by the diagonal of the Fisher information matrix F 1 at θ * 1. This gives the EWC loss the following form: DISPLAYFORM0 where L 2 (θ) = log p(D 2 |θ) is the loss for Task 2 individually, λ is a hyperparameter representing the importance of Task 1 relative to Task 2, DISPLAYFORM1 2, i is the parameter index, and L(θ) is the new loss to optimize while learning Task 2. Intuitively, the EWC loss prevents the model from straying too far away from the parameters important for Task 1 while leaving less crucial parameters free to model Task 2. Subsequent tasks in additional L EWC (θ) terms added to the loss for each previous task. By protecting the parameters deemed important for prior tasks, EWC as a regularization term allows a single neural network (assuming sufficient parameters and capacity) to learn new tasks in a sequential fashion, without forgetting how to perform previous tasks. While EWC makes a point estimate of how essential each parameter is at the of a task, IS BID39 protects the parameters according to their importance along the task's entire training trajectory. Termed synapses, each parameter θ i of the neural network is awarded an importance measure ω 1,i based on how much it reduced the loss while learning Task 1. Given a loss gradient g(t) = ∇ θ L(θ)| θ=θt at time t, the total change in loss during the training of Task 1 then is the sum of differential changes in loss over the training trajectory. With the assumption that parameters θ are independent, we have:where θ = dθ dt and (t 0, t 1) are the start and finish of Task 1, respectively. Note the added negative sign, as importance is associated with parameters that decrease the loss. The importance measure ω 1,i can now be used to introduce a regularization term that protects parameters important for Task 1 from large parameter updates, just as the Fisher information matrix diagonal terms F 1,i were used in EWC. This in an IS loss very reminiscent in form 1: DISPLAYFORM0 3.2 GAN CONTINUAL LEARNING The traditional continual learning methods are designed for certain canonical benchmarks, commonly consisting of a small number of clearly defined tasks (e.g., classification datasets in sequence). In GANs, the discriminator is trained on dataset DISPLAYFORM1 However, because of the evolution of the generator, the distribution p gen (x) from which D gen t comes changes over time. This violates the i.i.d. assumption of the order in which we present the discriminator data. As such, we argue that different instances in time of the generator should be viewed as separate tasks. Specifically, in the parlance of continual learning, the training data are to be regarded as DISPLAYFORM2 Thus motivated, we would like to apply continual learning methods to the discriminator, but doing so is not straightforward for the following reasons:• Definition of a task: EWC and IS were originally proposed for discrete, well-defined tasks. For example, BID15 applied EWC to a DQN BID25 learning to play ten Atari games sequentially, with each game being a clear, independent task. For GAN, there is no such precise definition as to what constitutes a "task," and as discriminators are not typically trained to convergence at every iteration, it is also unclear how long a task should be.• Computational memory: While Equations 2 and 4 are for two tasks, they can be extended to K tasks by adding a term L EWC or L IS for each of the K − 1 prior tasks. As each term L EWC or L IS requires saving both a historical reference term θ * k and either F k or ω k (all of which are the same size as the model parameters θ) for each task k, employing these techniques naively quickly becomes impractical for bigger models when K gets large, especially if K is set to the number of training iterations T.• Continual not learning: Early iterations of the discriminator are likely to be non-optimal, and without a forgetting mechanism, EWC and IS may forever lock the discriminator to a poor initialization. Additionally, the unconstrained addition of a large number of terms L EWC or L IS will cause the continual learning regularization term to grow unbounded, which can disincentivize any further changes in θ. To address these issues, we build upon the aforementioned continual learning techniques, and propose several changes. Number of tasks as a rate: We choose the total number of tasks K as a function of a constant rate α, which denotes the number of iterations before the of a task, as opposed to arbitrarily dividing the GAN training iterations into some set number of segments. Given T training iterations, this means a rate α yields K = T α tasks. Online Memory: Seeking a way to avoid storing extra θ * k, F k, or ω k, we observe that the sum of two or more quadratic forms is another quadratic, which gives the classifier loss with continual learning the following form for the (k + 1) th task: DISPLAYFORM3 DISPLAYFORM4 κ,i, and Q κ,i is either F κ,i or ω κ,i, depending on the method. We name models with EWC and IS augmentations EWC-GAN and IS-GAN, respectively.1 BID39 DISPLAYFORM5, where ∆1,i = θ1,i − θ0,i and ξ is a small number for numerical stability. We however found that the inclusion of (∆1,i) 2 can lead to the loss exploding and then collapsing as the number of tasks increases and so omit it. We also change the hyperparameter c into Under review as a conference paper at ICLR 2019Controlled forgetting: To provide a mechanism for forgetting earlier non-optimal versions of the discriminator and to keep L CL bounded, we add a discount factor γ: DISPLAYFORM6 Together, α and γ determine how far into the past the discriminator remembers previous generator distributions, and λ controls how important memory is relative to the discriminator loss. Note, the terms S k and P k can be updated every α steps in an online fashion: DISPLAYFORM7 This allows the EWC or IS loss to be applied without necessitating storing either Q k or θ * k for every task k, which would quickly become too costly to be practical. Only a single variable to store a running average is required for each of S k and P k, making this method space efficient. Augmenting the discriminator with the continual learning loss, the GAN objective becomes: DISPLAYFORM8 Note that the training of the generator remains the same; full algorithms are in Appendix A. Here we have shown two methods to mitigate catastrophic forgetting for the original GAN; however, the proposed framework is applicable to almost all of the wide range of GAN setups. Continual learning in GANs There has been previous work investigating continual learning within the context of GANs. Improved GAN BID32 introduced historical averaging, which regularizes the model with a running average of parameters of the most recent iterations. Simulated+Unsupervised training BID34 proposed replacing half of each minibatch with previous generator samples during training of the discriminator, as a generated sample at any point in time should always be considered fake. However, such an approach necessitates a historical buffer of samples and halves the number of current samples that can be considered. Continual Learning GAN BID33 applied EWC to GAN, as we have, but used it in the context of the class-conditioned generator that learns classes sequentially, as opposed to all at once, as we propose. BID36 independently reached a similar on catastrophic forgetting in GANs, but focused on gradient penalties and momentum on toy problems. The heart of continual learning is distilling a network's knowledge through time into a single network, a temporal version of the ensemble described in BID10. There have been several proposed models utilizing multiple generators BID4 or multiple discriminators BID2 BID27, while Bayesian GAN BID31 considered distributions on the parameters of both networks, but these all do not consider time as the source of the ensemble. Unrolled GAN BID22 ) considered multiple discriminators "unrolled" through time, which is similar to our method, as the continual learning losses also utilize historical instances of discriminators. However, both EWC-GAN and IS-GAN preserve the important parameters for prior discriminator performance, as opposed to requiring backpropagation of generator samples through multiple networks, making them easier to implement and train. GAN convergence While GAN convergence is not the focus of this paper, convergence does similarly avoid mode collapse, and there are a number of works on the topic BID9 BID37 BID26 BID21. From the perspective of BID9, EWC or IS regularization in GAN can be viewed as achieving convergence by slowing the discriminator, but per parameter, as opposed to a slower global learning rate. 5.1 DISCRIMINATOR CATASTROPHIC FORGETTING While FIG0 implies catastrophic forgetting in a GAN discriminator, we can show this concretely. To do so, we first train a DCGAN on the MNIST dataset. Since the generator is capable of generating an arbitrary number of samples at any point, we can randomly draw 70000 samples to comprise a new, "fake MNIST" dataset at any time. By doing this at regular intervals, we create datasets {D Having previously generated a series of datasets during the training of a DCGAN, we now reinitialize the discriminator and train to convergence on each D after fine-tuning on D gen t ; this is unsurprising, as p gen (x) has evolved specifically to deteriorate discriminator performance. While there is still a dropoff with EWC, forgetting is less severe. While the training outlined above is not what is typical for GAN, we choose this set-up as it closely mirrors the continual learning literature. With recent criticisms of some common continual learning benchmarks as either being too easy or missing the point of continual learning BID3, we propose GAN as a new benchmark providing a more realistic setting. From FIG3, it is clear that while EWC certainly helps, there is still much room to improve with new continual learning methods. However, the merits of GAN as a continual learning benchmark go beyond difficulty. While it is unclear why one would ever use a single model to classify successive random permutations of MNIST , many real-world settings exist where the data distribution is slowly evolving. For such models, we would like to be able to update the deployed model without forgetting previously learned performance, especially when data collection is expensive and thus done in bulk sometime before deployment. For example, autonomous vehicles BID12 will eventually encounter unseen car models or obstacles, and automated screening systems at airport checkpoints BID18 will have to deal with evolving bags, passenger belongings, and threats. In both cases, sustained effectiveness requires a way to appropriately and efficiently update the models for new data, or risk obsolescence leading to dangerous blindspots. Many machine learning datasets represent singe-time snapshots of the data distribution, and current continual learning benchmarks fail to capture the slow drift of the real-world data. The evolution of GAN synthesized samples represents an opportunity to generate an unlimited number of smoothly evolving datasets for such experiments. We note that while the setup used here is for binary real/fake classification, one could also conceivably use a conditional GAN BID23 to generate an evolving multi-class classification dataset. We leave this exploration for future work. We show on a toy dataset consisting of a mixture of eight Gaussians, as in the example in FIG0. Following the setup of BID22, the real data are evenly distributed among eight 2-dimensional Gaussian distributions arranged in a circle of radius 2, each with covariance 0.02I (see Figure 4). We evaluate our model with Inception Score (ICP) BID32, which gives a rough measure of diversity and quality of samples; higher scores imply better performance, with the true data ing in a score of around 7.870. For this simple dataset, since we know the true data distribution, we also calculate the symmetric Kullback-Leibler divergence (Sym-KL); lower scores mean the generated samples are closer to the true data. We show computation time, measured in numbers of training iterations per second (Iter/s), averaged over the full training of a model on a single Nvidia Titan X (Pascal) GPU. Each model was run 10 times, with the mean and standard deviation of each performance metric at the end of 25K iterations reported in TAB1. The performance of EWC-GAN and IS-GAN were evaluated for a number of hyperparameter settings. We compare our against a vanilla GAN , as well as a state-ofthe-art GAN with spectral normalization (SN) BID24 applied to the discriminator. As spectral normalization augments the discriminator loss in a way different from continual learning, we can combine the two methods; this variant is also shown. Note that a discounted version of discriminator historical averaging BID32 can be recovered from the EWC and IS losses if the task rate α = 1 and Q k,i = 1 for all i and k, a poor approximation to both the Fisher information matrix diagonal and importance measure. If we also set the historical reference termθ * k and the discount factor γ to zero, then the EWC and IS losses become 2 weight regularization. These two special cases are also included for comparison. We observe that augmenting GAN models with EWC and IS consistently in generators that better match the true distribution, both qualitatively and quantitatively, for a wide range of hyperparameter settings. EWC-GAN and IS-GAN in a better ICP and FID than 2 weight regularization and discounted historical averaging, showing the value of prioritizing protecting important parameters, rather than all parameters equally. EWC-GAN and IS-GAN also outperform a stateof-the-art method in SN-GAN. In terms of training time, updating the EWC loss requires forward propagating a new minibatch through the discriminator and updating S and P, but even if this is done at every step (α = 1), the ing algorithm is only slightly slower than SN-GAN. Moreover, doing so is unnecessary, as higher values of α also provide strong performance for a much smaller time penalty. Combining EWC with SN-GAN leads to even better , showing that the two methods can complement each other. IS-GAN can also be successfully combined with SN-GAN, but it is slower than EWC-GAN as it requires tracking the trajectory of parameters at each step. Sample generation evolution over time is shown in Figure 4 of Appendix C. Since EWC-GAN achieves similar performance to IS-GAN but at less computational expense, we focus on the former for experiments on two image datasets, CelebA and CIFAR-10. Our EWC-GAN implementation is straightforward to add to any GAN model, so we augment various popular implementations. Comparisons are made with the TTUR BID9 variants 2 of DCGAN and WGAN-GP BID7, as well as an implementation 3 of a spectral normalized BID24 ) DCGAN (SN-DCGAN). Without modifying the learning rate or model architecture, we show with and without the EWC loss term added to the discriminator for each. Performance is quantified with the Fréchet Inception Distance (FID) BID9 for both datasets. Since labels are available for CIFAR-10, we also report ICP for that dataset. Best values are reported in TAB2, with samples in Appendix C. In each model, we see improvement in both FID and ICP from the addition of EWC to the discriminator. We also consider the text generation on the MS COCO Captions dataset BID1, with the pre-processing in BID8. Quality of generated sentences is evaluated by BLEU score BID28. Since BLEU-b measures the overlap of b consecutive words between the generated sentences and ground-truth references, higher BLEU scores indicate better fluency. Self BLEU uses the generated sentences themselves as references; lower values indicate higher diversity. We apply EWC and IS to textGAN, a recently proposed model for text generation in which the discriminator uses feature matching to stabilize training. This model's (labeled "EWC" and "IS") are compared to a Maximum Likelihood Estimation (MLE) baseline, as well as several state-of-the-art methods: SeqGAN BID38, RankGAN BID19, GSGAN BID13 and LeakGAN . Our variants of textGAN outperforms the vanilla textGAN for all BLEU scores (see TAB3), indicating the effectiveness of addressing the forgetting issue for GAN training in text generation. EWC/IS + textGAN also demonstrate a significant improvement compared with other methods, especially on BLEU-2 and 3. Though our variants lag slightly behind LeakGAN on BLEU-4 and 5, their self BLEU scores TAB4 indicate it generates more diverse sentences. Sample sentence generations can be found in Appendix C. We observe that the alternating training procedure of GAN models in a continual learning problem for the discriminator, and training on only the most recent generations leads to consequences unaccounted for by most models. As such, we propose augmenting the GAN training objective with a continual learning regularization term for the discriminator to prevent its parameters from moving too far away from values that were important for recognizing synthesized samples from previous training iterations. Since the original EWC and IS losses were proposed for discrete tasks, we adapt them to the GAN setting. Our implementation is simple to add to almost any variation of GAN learning, and we do so for a number of popular models, showing a gain in ICP and FID for CelebA and CIFAR-10, as well as BLEU scores for COCO Captions. More importantly, we demonstrate that GAN and continual learning, two popular fields studied independently of each other, have the potential to benefit each other, as new continual learning methods stand to benefit GAN training, and GAN generated datasets provide new testing grounds for continual learning. To produce a smoothly evolving series of datasets for continual learning, we train a DCGAN on MNIST and generate an entire "fake" dataset of 70K samples every 50 training iterations of the DC-GAN generator. We propose learning each of these generated datasets as individual tasks for continual learning. Selected samples are shown in Figure 3 from the datasets D gen t for t ∈ {5, 10, 15, 20}, each generated from the same 100 samples of z for all t. Note that we actually trained a conditional DCGAN, meaning we also have the labels for each generated image. For experiments in FIG3, we focused on the real versus fake task to demonstrate catastrophic forgetting in a GAN discriminator and thus ignored the labels, but future experiments can incorporate such information. Figure 3: Image samples from generated "fake MNIST" datasets C EXAMPLES OF GENERATED SAMPLES Sample generations are plotted during training at 5000 step intervals in Figure 4. While vanilla GAN occasionally recovers the true distribution, more often than not, the generator collapses and then bounces around. Spectral Normalized GAN converges to the true distribution quickly in most runs, but it mode collapses and exhibits the same behavior as GAN in others. EWC-GAN consistently diffuses to all modes, tending to find the true distribution sooner with lower α. We omit IS-GAN, as it performs similarly to EWC-GAN. Figure 4: Each row shows the evolution of generator samples at 5000 training step intervals for GAN, SN-GAN, and EWC-GAN for two α values. The proposed EWC-GAN models have hyperparameters matching the corresponding α in TAB1. Each frame shows 10000 samples drawn from the true eight Gaussians mixture (red) and 10000 generator samples (blue). We also show the generated image samples for CIFAR 10 and CelebA in Figure 5, and generated text samples for MS COCO Captions in Table 5.(a) CIFAR 10 (b) CelebA Figure 5: Generated image samples from random draws of EWC+GANs. Table 5: Sample sentence generations from EWC + textGAN a couple of people are standing by some zebras in the the view of some benches near a gas station a brown motorcycle standing next to a red fence a bath room with a broken tank on the floor red passenger train parked under a bridge near a river some snow on the beach that is surrounded by a truck a cake that has been perform in the for takeoff a view of a city street surrounded by trees two giraffes walking around a field during the day crowd of people lined up on motorcycles two yellow sheep with a baby dog in front of other sheep an intersection sits in front of a crowd of people a red double decker bus driving down the street corner an automobile driver stands in the middle of a snowy park five people at a kitchen setting with a woman there are some planes at the takeoff station a passenger airplane flying in the sky over a cloudy sky three aircraft loaded into an airport with a stop light there is an animal walking in the water an older boy with wine glasses in an office two old jets are in the middle of london three motorcycles parked in the shade of a crowd group of yellow school buses parked on an intersection a person laying on a sidewalk next to a sidewalk talking on a cell phone a chef is preparing food with a sink and stainless steel appliances

Represent the TLDR for finding the corresponding scientific article

Generative Adversarial Network Training is a Continual Learning Problem.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.