Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions. We analyze the convergence of GAN training from this new point of view to understand why mode collapse happens. We hypothesize the existence of undesirable local equilibria in this non-convex game to be responsible for mode collapse. We observe that these local equilibria often exhibit sharp gradients of the discriminator function around some real data points. We demonstrate that these degenerate local equilibria can be avoided with a gradient penalty scheme called DRAGAN. We show that DRAGAN enables faster training, achieves improved stability with fewer mode collapses, and leads to generator networks with better modeling performance across a variety of architectures and objective functions. Generative modeling involves taking a set of samples drawn from an unknown data generating distribution P real and finding an estimate P model that closely resembles it. Generative adversarial networks (GAN) BID6 ) is a powerful framework used for fitting implicit generative models. The basic setup consists of two networks, the generator and the discriminator, playing against each other in a repeated zero-sum game setting. The goal here is to reach an equilibrium where P real, P model are close, and the alternating gradient updates procedure (AGD) is used to achieve this. However, this process is highly unstable and often in mode collapse BID7. This calls for an deeper investigation into training dynamics of GANs. In this paper, we propose studying GAN training dynamics as a repeated game in which both the players are using no-regret algorithms BID2 and discuss how AGD 1 falls under this paradigm. In contrast, much of the theory BID6 BID0 and recent developments BID15 BID8 are based on the unrealistic assumption that the discriminator is playing optimally (in the function space) at each step and as a , there is consistent minimization of a divergence between real and generated distributions. This corresponds to at least one player using the best-response algorithm (in the function space), and the ing game dynamics can be completely different in both these cases BID14. Thus, there is a clear disconnect between theoretical arguments used as motivation in recent literature and what actually happens in practice. We would like to point out that the latter view can still be useful for reasoning about the asymptotic equilibrium situation but we argue that regret minimization is the more appropriate way to think about GAN training dynamics. So, we analyze the convergence of GAN training from this new point of view to understand why mode collapse happens. We start with a short analysis of the artificial convex-concave case of the GAN game in section 2.2. This setting has a unique solution and guaranteed convergence (of averaged iterates) using no-regret algorithms can be shown with standard arguments from game theory literature. Here, we make explicit, the critical (previously not widely known) connection between AGD used in GAN training and regret minimization. This immediately yields a novel proof for the asymptotic convergence of GAN training, in the non-parametric limit. Prior to our work, such a BID6 ) required a strong assumption that the discriminator is optimal at each step. However, these convergence do not hold when the game objective function is non-convex, which is the practical case when deep neural networks are used. In non-convex games, global regret minimization and equilibrium computation are computationally hard in general. Recent gametheoretic literature indicates that AGD can end up cycling BID11 or converging to a (potentially bad) local equilibrium, under some conditions BID9. We hypothesize these to be the reasons for cycling and mode collapse observed during GAN training, respectively (section 2.3). In this work, we do not explore the cycling issue but focus our attention on the mode collapse problem. In contrast to our hypothesis, the prevalent view of mode collapse and instability BID0 is that it from attempting to minimize a strong divergence during training. However, as we argued earlier, GAN training with AGD does not consistently minimize a divergence and therefore, such a theory is not suitable to discuss convergence or to address the stability issue. Next, if mode collapse is indeed the of an undesirable local equilibrium, a natural question then is how we can avoid it? We make a simple observation that, in the GAN game, mode collapse situations are often accompanied by sharp gradients of the discriminator function around some real data points (section 2.4). Therefore, a simple strategy to mitigate mode collapse is to regularize the discriminator so as to constrain its gradients in the ambient data space. We demonstrate that this improves the stability using a toy experiment with one hidden layer neural networks. This gives rise to a new explanation for why WGAN and gradient penalties might be improving the stability of GAN training -they are mitigating the mode collapse problem by keeping the gradients of the discriminator function small in data space. From this motivation, we propose a training algorithm involving a novel gradient penalty scheme called DRAGAN (Deep Regret Analytic Generative Adversarial Networks) which enables faster training, achieves improved stability and modeling performance (over WGAN-GP BID8 which is the state-of-the-art stable training procedure) across a variety of architectures and objective functions. Below, we provide a short literature review. Several recent works focus on stabilizing the training of GANs. While some solutions BID17 BID18 require the usage of specific architectures (or) modeling objectives, some BID4 BID20 significantly deviate from the original GAN framework. Other promising works in this direction BID12 BID16 BID8 ) impose a significant computational overhead. Thus, a fast and versatile method for consistent stable training of GANs is still missing in the literature. Our work is aimed at addressing this. To summarize, our contributions are as follows:• We propose a new way of reasoning about the GAN training dynamics -by viewing AGD as regret minimization.• We provide a novel proof for the asymptotic convergence of GAN training in the nonparametric limit and it does not require the discriminator to be optimal at each step.• We discuss how AGD can converge to a potentially bad local equilibrium in non-convex games and hypothesize this to be responsible for mode collapse during GAN training.• We characterize mode collapse situations with sharp gradients of the discriminator function around some real data points.• A novel gradient penalty scheme called DRAGAN is introduced based on this observation and we demonstrate that it mitigates the mode collapse issue. We start with a brief description of the GAN framework (section 2.1). We discuss guaranteed convergence in the artificial convex-concave case using no-regret algorithms, and make a critical connection between GAN training process (AGD) and regret minimization (section 2.2). This immediately yields a novel proof for the asymptotic convergence of GAN training in the nonparametric limit. Then, we consider the practical non-convex case and discuss how AGD can converge to a potentially bad local equilibrium here (section 2.3). We characterize mode collapse situations with sharp gradients of the discriminator function around real samples and this provides an effective strategy to avoid them. This naturally leads to the introduction of our gradient penalty scheme DRAGAN (section 2.4). We end with a discussion and comparison with other gradient penalties in the literature (section 2.5). The GAN framework can be viewed as a repeated zero-sum game, consisting of two players -the generator, which produces synthetic data given some noise source and the discriminator, which is trained to distinguish generator's samples from the real data. The generator model G is parameterized by φ, takes a noise vector z as input, and produces a synthetic sample G φ (z). The discriminator model D is parameterized by θ, takes a sample x as input and computes D θ (x), which can be interpreted as the probability that x is real. The models G, D can be selected from any arbitrary class of functions -in practice, GANs typical rely on deep networks for both. Their cost functions are defined as DISPLAYFORM0, and DISPLAYFORM1 And the complete game can be specified as - DISPLAYFORM2 The generator distribution P model asymptotically converges to the real distribution P real if updates are made in the function space and the discriminator is optimal at each step BID6. According to Sion's theorem BID19, if Φ ⊂ R m, Θ ⊂ R n such that they are compact and convex sets, and the function J: Φ × Θ → R is convex in its first argument and concave in its second, then we havemin DISPLAYFORM0 That is, an equilibrium is guaranteed to exist in this setting where players' payoffs correspond to the unique value of the game BID13.A natural question then is how we can find such an equilibrium. A simple procedure that players can use is best-response algorithms (BRD). In each round, best-responding players play their optimal strategy given their opponent's current strategy. Despite its simplicity, BRD are often computationally intractable and they don't lead to convergence even in simple games. In contrast, a technique that is both efficient and provably works is regret minimization. If both players update their parameters using no-regret algorithms, then it is easy to show that their averaged iterates will converge to an equilibrium pair BID14. Let us first define no-regret algorithms. DISPLAYFORM1, where we define DISPLAYFORM2 We can apply no-regret learning to our problem of equilibrium finding in the GAN game J(·, ·) as follows. The generator imagines the function J(·, θ t) as its loss function on round t, and similarly the discriminator imagines −J(φ t, ·) as its loss function at t. After T rounds of play, each player computes the average iteratesφ DISPLAYFORM3 If V * is the equilibrium value of the game, and the players suffer regret R 1 (T) and R 2 (T) respectively, then one can show using standard arguments BID5 ) that - DISPLAYFORM4 T. In other words,θ T andφ T are "almost optimal" solutions to the game, where the "almost" approximation factor is given by the average regret terms DISPLAYFORM5. Under the no-regret condition, the former will vanish, and hence we can guarantee convergence in the limit. Next, we define a popular family of no-regret algorithms. Definition 2.2 (Follow The Regularized Leader). FTRL BID10 selects k t on round t by solving for arg min DISPLAYFORM6 where Ω(·) is some convex regularization function and η is a learning rate. Remark: Roughly speaking, if you select the regularization as Ω(·) = 1 2 · 2, then FTRL becomes the well-known online gradient descent or OGD BID21. Ignoring the case of constraint violations, OGD can be written in a simple iterative form: DISPLAYFORM7 The typical GAN training procedure using alternating gradient updates (or simultaneous gradient updates) is almost this -both the players applying online gradient descent. Notice that the min/max objective function in GANs involves a stochastic component, with two randomized inputs given on each round, x and z which are sampled from the data distribution and a standard multivariate normal, respectively. Let us write DISPLAYFORM8 Taking expectations with respect to x and z, we define the full (non-stochastic) game as DISPLAYFORM9 But the above online training procedure is still valid with stochastic inputs. That is, the equilibrium computation would proceed similarly, where on each round we sample x t and z t, and follow the updates DISPLAYFORM10 On a side note, a benefit of this stochastic perspective is that we can get a generalization bound on the mean parametersφ T after T rounds of optimization. The celebrated "online-to-batch conversion" BID3 implies that E x,z [J x,z (φ T, θ)], for any θ, is no more than the optimal value DISPLAYFORM11, where the expectation is taken with respect to the sequence of samples observed along the way, and any randomness in the algorithm. Analogously, this applies toθ T as well. A limitation of this , however, is that it requires a fresh sample x t to be used on every round. To summarize, we discussed in this subsection about how the artificial convex-concave case is easy to solve through regret minimization. While this is a standard in game theory and online learning literature, it is not widely known in the GAN literature. For instance, BID18 and BID7 discuss a toy game which is convex-concave and show cycling behavior. But, the simple solution in that case is to just average the iterates. Further, we made explicit, the critical connection between regret minimization and alternating gradient updates procedure used for GAN training. Now, BID6 argue that, if G and D have enough capacity (in the non-parametric limit) and updates are made in the function space, then the GAN game can be considered convex-concave. Thus, our analysis based on regret minimization immediately yields a novel proof for the asymptotic convergence of GANs, without requiring that the discriminator be optimal at each step. Moreover, the connection between regret minimization and GAN training process gives a novel way to reason about its dynamics. In contrast, the popular view of GAN training as consistently minimizing a divergence arises if the discriminator uses BRD (in the function space) and thus, it has little to do with the actual training process of GANs. As a , this calls into question the motivation behind many recent developments like WGAN and gradient penalties among others, which improve the training stability of GANs. In the next subsection, we discuss the practical non-convex case and why training instability arises. This provides the necessary ideas to investigate mode collapse from our new perspective. In practice, we choose G, D to be deep neural networks and the function J(φ, θ) need not be convexconcave anymore. The nice properties we had in the convex-concave case like the existence of a unique solution and guaranteed convergence through regret minimization no longer hold. In fact, regret minimization and equilibrium computation are computationally hard in general non-convex settings. However, analogous to the case of non-convex optimization (also intractable) where we focus on finding local minima, we can look for tractable solution concepts in non-convex games. Recent work by BID9 introduces the notion of local regret and shows that if both the players use a smoothed variant of OGD to minimize this quantity, then the non-convex game converges to some form of local equilibrium, under mild assumptions. The usual training procedure of GANs (AGD) corresponds to using a window size of 1 in their formulation. Thus, GAN training will eventually converge (approximately) to a local equilibrium which is described below or the updates will cycle. We leave it to future works to explore the equally important cycling issue and focus here on the former case. DISPLAYFORM0 That is, in a local equilibrium, both the players do not have much of an incentive to switch to any other strategy within a small neighborhood of their current strategies. Now, we turn our attention to the mode collapse issue which poses a significant challenge to the GAN training process. The training is said to have ed in mode collapse if the generator ends up mapping multiple z vectors to the same output x, which is assigned a high probability of being real by the discriminator BID7. We hypothesize this to be the of the game converging to bad local equilibria. The prevalent view of mode collapse and instability in GAN training BID0 is that it is caused due to the supports of real and model distributions being disjoint or lying on low-dimensional manifolds. The argument is that this would in strong distance measures like KL-divergence or JS-divergence getting maxed out, and the generator cannot get useful gradients to learn. In fact, this is the motivation for the introduction of WGAN. But, as we argued earlier, GAN training does not consistently minimize a divergence as that would require using intractable best-response algorithms. Hence, such a theory is not suitable to discuss convergence or to address the instability of GAN training. Our new view of GAN training process as regret minimization is closer to what is used in practice and provides an alternate explanation for mode collapse -the existence of undesirable local equilibria. The natural question now is how we can avoid them? The problem of dealing with multiple equilibria in games and how to avoid undesirable ones is an important question in algorithmic game theory BID14. In this work, we constrain ourselves to the GAN game and aim to characterize the undesirable local equilibria (mode collapse) in an effort to avoid them. In this direction, after empirically studying multiple mode collapse cases, we found that it is often accompanied by the discriminator function having sharp gradients around some real data points (See FIG0). This intuitively makes sense from the definition of mode collapse discussed earlier. Such sharp gradients encourage the generator to map multiple z vectors to a single output x and lead the game towards a degenerate equilibrium. Now, a simple strategy to mitigate this failure case would be to regularize the discriminator using the following penalty - DISPLAYFORM0 This strategy indeed improves the stability of GAN training. We show the of a toy experiment with one hidden layer neural networks in FIG0 and FIG1 to demonstrate this. This partly explains the success of WGAN and gradient penalties in the recent literature BID8 BID16, and why they improve the training stability of GANs, despite being motivated by reasoning based on unrealistic assumptions. However, we noticed that this scheme in its current form can be brittle and if over-penalized, the discriminator can end up assigning both a real point x and noise x + δ, the same probability of being real. Thus, a better choice of penalty is - DISPLAYFORM1 Finally, due to practical optimization considerations (this has also been observed in BID8), we instead use the penalty shown below in all our experiments. DISPLAYFORM2 2 At times, stochasticity seems to help in getting out of the basin of attraction of a bad equilibrium Figure 1: One hidden layer networks as G and D (MNIST). On the left, we plot inception score against time for vanilla GAN training and on the right, we plot the squared norm of discriminator's gradients around real data points for the same experiment. Notice how this quantity changes before, during and after mode collapse events. This still works as long as small perturbations of real data, x + δ are likely to lie off the data-manifold, which is true in the case of image domain and some other settings. Because, in these cases, we do want our discriminator to assign different probabilities of being real to training data and noisy samples. We caution the practitioners to keep this important point in mind while making their choice of penalty. All of the above schemes have the same effect of constraining the norm of discriminator's gradients around real points to be small and can therefore, mitigate the mode collapse situation. We refer to GAN training using these penalty schemes or heuristics as the DRAGAN algorithm. Additional details:• We use the vanilla GAN objective in our experiments, but our penalty improves stability using other objective functions as well. This is demonstrated in section 3.3.• The penalty scheme used in our experiments is the one shown in equation 1.• We use small pixel-level noise but it is possible to find better ways of imposing this penalty. However, this exploration is beyond the scope of our paper.• The optimal configuration of the hyperparameters for DRAGAN depends on the architecture, dataset and data domain. We set them to be λ ∼ 10, k = 1 and c ∼ 10 in most of our experiments. Several recent works have also proposed regularization schemes which constrain the discriminator's gradients in the ambient data space, so as to improve the stability of GAN training. Despite being from different motivations, WGAN-GP and LS-GAN are closely related approaches to ours. First, we show that these two approaches are very similar, which is not widely known in the literature. BID16 introduced LS-GAN with the idea of maintaining a margin between losses assigned to real and fake samples. Further, they also impose Lipschitz constraint on D and the two conditions together in a situation where the following holds for any real and fake sample pair (roughly) - DISPLAYFORM0 The authors argue that the ing discriminator function would have non-vanishing gradients almost everywhere between real and fake samples (section 6 of Qi FORMULA18). Next, BID8 proposed an extension to address various shortcomings of the original WGAN and they impose the following condition on D - DISPLAYFORM1 is some point on the line between a real and a fake sample, both chosen independently at random. This leads to D having norm-1 gradients almost everywhere between real and fake samples. Notice that this behavior is very similar to that of LS-GAN's discriminator function. Thus, WGAN-GP is a slight variation of the original LS-GAN algorithm and we refer to these methods as "coupled penalties".On a side note, we also want to point out that WGAN-GP's penalty doesn't actually follow from KR-duality as claimed in their paper. By Lemma 1 of BID8, the optimal discriminator D * will have norm-1 gradients (almost everywhere) only between those x and G φ (z) pairs which are sampled from the optimal coupling or joint distribution π *. Therefore, there is no basis for WGAN-GP's penalty (equation 3) where arbitrary pairs of real and fake samples are used. This fact adds more credence to our theory regarding why gradient penalties might be mitigating mode collapse. The most important distinction between coupled penalties and our methods is that we only impose gradient constraints in local regions around real samples. We refer to these penalty schemes as "local penalties". Coupled penalties impose gradient constraints between real and generated samples and we point out some potential issues that arise from this:• With adversarial training finding applications beyond fitting implicit generative models, penalties which depend on generated samples can be prohibitive.• The ing class of functions when coupled penalties are used will be highly restricted compared to our method and this affects modeling performance. We refer the reader to FIG2 and appendix section 5.2.2 to see this effect.• Our algorithm works with AGD, while WGAN-GP needs multiple inner iterations to optimize D. This is because the generated samples can be anywhere in the data space and they change from one iteration to the next. In contrast, we consistently regularize D θ (x) only along the real data manifold. To conclude, appropriate constraining of the discriminator's gradients can mitigate mode collapse but we should be careful so that it doesn't have any negative effects. We pointed out some issues with coupled penalties and how local penalties can help. We refer the reader to section 3 for further experimental . In section 3.1, we compare the modeling performance of our algorithm against vanilla GAN and WGAN variants in the standard DCGAN/CIFAR-10 setup. Section 3.2 demonstrates DRAGAN's improved stability across a variety of architectures. In section 3.3, we show that our method also works with other objective functions. Appendix contains samples for inspection, some of the missing plots and additional . Throughout, we use inception score BID18 which is a well-studied and reliable metric in the literature, and sample quality to measure the performance. DCGAN is a family of architectures designed to perform well with the vanilla training procedure. They are ubiquitous in the GAN literature owing to the instability of vanilla GAN in general settings. We use this architecture to model CIFAR-10 and compare against vanilla GAN, WGAN and WGAN-GP. As WGANs need 5 discriminator iterations for every generator iteration, comparing the modeling performance can be tricky. To address this, we report two scores for vanilla GAN and DRAGANone using the same number of generator iterations as WGANs and one using the same number of discriminator iterations. The are shown in FIG3 and samples are included in the appendix (Figure 8). Notice that DRAGAN beats WGAN variants in both the configurations, while vanilla GAN is only slightly better. A key point to note here is that our algorithm is fast compared to WGANs, so in practice, the performance will be closer to the DRAGAN d case. In the next section, we will show that if we move away from this specific architecture family, vanilla GAN training can become highly unstable and that DRAGAN penalty mitigates this issue. Ideally, we would want our training procedure to perform well in a stable fashion across a variety of architectures (other than DCGANs). Similar to and BID8, we remove the stabilizing components of DCGAN architecture and demonstrate improved stability & modeling performance compared to vanilla GAN training (see appendix section 5.2.3). However, this is a small set of architectures and it is not clear if there is an improvement in general. To address this, we introduce a metric termed the BogoNet score to compare the stability & performance of different GAN training procedures. The basic idea is to choose random architectures for players G and D independently, and evaluate the performance of different algorithms in the ing games. A good algorithm should achieve stable performance without failing to learn or ing in mode collapse, despite the potentially imbalanced architectures. In our experiment, each player is assigned a network from a diverse pool of architectures belonging to three different families (MLP, ResNet, DCGAN). To demonstrate that our algorithm performs better compared to vanilla GAN training and WGAN-GP, we created 100 such instances of hard games. Each instance is trained using these algorithms on CIFAR-10 (under similar conditions for a fixed number of generator iterations, which gives a slight advantage to WGAN-GP) and we plot how inception score changes over time. For each algorithm, we calculated the average of final inception scores and area under the curve (AUC) over all 100 instances. The are shown in TAB0. Notice that we beat the other algorithms in both metrics, which indicates some improvement in stability and modeling performance. Further, we perform some qualitative analysis to verify that BogoNet score indeed captures the improvements in stability. We create another set of 50 hard architectures and compare DRAGAN against vanilla GAN training. Each instance is allotted 5 points and we split this bounty between the two algorithms depending on their performance. If both perform well or perform poorly, they get 2.5 points each, so that we nullify the effect of such non-differentiating architectures. However, if one algorithm achieves stable performance compared to the other (in terms of failure to learn or mode collapses), we assign it higher portions of the bounty. Results were judged by two of the authors in a blind manner: The curves were shown side-by-side with the choice of algorithm for each side being randomized and unlabeled. The vanilla GAN received an average score of 92.5 while our algorithm achieved an average score of 157.5 and this correlates with BogoNet score from earlier. See appendix section 5.3 for some additional details regarding this experiment. Our algorithm improves stability across a variety of objective functions and we demonstrate this using the following experiment. BID15 show that we can interpret GAN training as minimizing various f -divergences when an appropriate game objective function is used. We show experiments using the objective functions developed for Forward KL, Reverse KL, Pearson χ 2, Squared Hellinger, and Total Variation divergence minimization. We use a hard architecture from the previous subsection to demonstrate the improvements in stability. Our algorithm is stable in all cases except for the total variation case, while the vanilla algorithm failed in all the cases (see FIG4 for two examples and FIG3 in appendix for all five). Thus, practitioners can now choose their game objective from a larger set of functions and use DRAGAN (unlike WGANs which requires a specific objective function). In this paper, we propose to study GAN training process as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions. We analyze the convergence of GAN training from this new point of view and hypothesize that mode collapse occurs due to the existence of undesirable local equilibria. A simple observation is made about how the mode collapse situation often exhibits sharp gradients of the discriminator function around some real data points. This characterization partly explains the workings of previously proposed WGAN and gradient penalties, and motivates our novel penalty scheme. We show evidence of improved stability using DRAGAN and the ing improvements in modeling performance across a variety of settings. We leave it to future works to explore our ideas in more depth and come up with improved training algorithms. In this section, we provide samples from an additional experiment run on CelebA dataset (Figure 7). The samples from the experiment in section 3.1 are shown in Figure 8. Further, BID17 suggest that walking on the manifold learned by the generator can expose signs of memorization. We use DCGAN architecture to model MNIST and CelebA datasets using DRAGAN penalty, and the latent space walks of the learned models are shown in Figure 9 and Figure 10. The demonstrate that the generator is indeed learning smooth transitions between different images, when our algorithm is used. We design a simple experiment where G and D are both fully connected networks with just one hidden layer. Vanilla GAN performs poorly even in this simple case and we observe severe mode collapses. In contrast, our algorithm is stable throughout and obtains decent quality samples despite the constrained setup. We analyze the performance of WGAN-GP and DRAGAN on the 8-Gaussians dataset. As it can be seen in FIG1, both of them approximately converge to the real distribution but notice that in the case of WGAN-GP, D θ (x) seems overly constrained in the data space. In contrast, DRAGAN's discriminator is more flexible. Orange is real samples, green is generated samples. The level sets of D θ (x) are shown in the , with yellow as high and purple as low. DCGAN architecture has been designed following specific guidelines to make it stable BID17. We restate the suggested rules here.1. Use all-convolutional networks which learn their own spatial downsampling (discriminator) or upsampling (generator) 2. Remove fully connected hidden layers for deeper architectures 3. Use batch normalization in both the generator and the discriminator 4. Use ReLU activation in the generator for all layers except the output layer, which uses tanh We show below that such constraints can be relaxed when using our algorithm and still maintain training stability. Below, we present a series of experiments in which we remove different stabilizing components from the DCGAN architecture and analyze the performance of our algorithm. Specifically, we choose the following four architectures which are difficult to train (in each case, we start with base DCGAN architecture and apply the changes) -• No BN and a constant number of filters in the generator• 4-layer 512-dim ReLU MLP generator• tanh nonlinearities everywhere• tanh nonlinearity in the generator and 4-layer 512-dim LeakyReLU MLP discriminator Notice that, in each case, our algorithm is stable while the vanilla GAN training fails. A similar approach is used to demonstrate the stability of training procedures in and BID8. Due to space limitations, we only showed plots for two cases in section 3.3. Below we show the for all five cases. We used three families of architectures with probabilities -DCGAN (0.6), ResNet (0.2), MLP (0.2). Next, we further parameterized each family to create additional variation. For instance, the DCGAN family can in networks with or without batch normalization, have LeakyReLU or Tanh nonlinearities. The number and width of filters, latent space dimensionality are some other possible variations in our experiment. Similarly, the number of layers and hidden units in each layer for MLP are chosen randomly. For ResNets, we chose their depth randomly. This creates a set of hard games which test the stability of a given training algorithm. We showed qualitative analysis of the inception score plots in section 3.2 to verify that BogoNet score indeed captures the improvements in stability. Below, we show some examples of how the bounty splits were done. The plots in FIG2 were scored as (averages are shown in DRAGAN, Vanilla GAN order):A -, B -(3.5, 1.5), C -(2.25, 2.75), D -

Analysis of convergence and mode collapse by studying GAN training process as regret minimization

Represent the TLDR for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep neural networks (DNNs) have attained surprising achievement during the last decade due to the advantages of automatic feature learning and freedom of expressiveness. However, their interpretability remains mysterious because DNNs are complex combinations of linear and nonlinear transformations. Even though many models have been proposed to explore the interpretability of DNNs, several challenges remain unsolved: 1) The lack of interpretability quantity measures for DNNs, 2) the lack of theory for stability of DNNs, and 3) the difficulty to solve nonconvex DNN problems with interpretability constraints. To address these challenges simultaneously, this paper presents a novel intrinsic interpretability evaluation framework for DNNs. Specifically, Four independent properties of interpretability are defined based on existing works. Moreover, we investigate the theory for the stability of DNNs, which is an important aspect of interpretability, and prove that DNNs are generally stable given different activation functions. Finally, an extended version of deep learning Alternating Direction Method of Multipliers (dlADMM) are proposed to solve DNN problems with interpretability constraints efficiently and accurately. Extensive experiments on several benchmark datasets validate several DNNs by our proposed interpretability framework.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel framework to evaluate the interpretability of neural network.

We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design, with theoretical guarantees.


Question: represent the article for finding a single-sentence tldr.

Question: Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. Here we extend earlier work of Kurth-Nelson and Redish and propose an efficient deep reinforcement learning agent that acts via hyperbolic discounting and other non-exponential discount mechanisms. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over state-of-the-art methods. The standard treatment of the reinforcement learning (RL) problem is the Markov Decision Process (MDP) which includes a discount factor 0 ≤ γ ≤ 1 that exponentially reduces the present value of future rewards . A reward r t received in t-time steps is devalued to γ t r t, a discounted utility model introduced by. This establishes a timepreference for rewards realized sooner rather than later. The decision to exponentially discount future rewards by γ leads to value functions that satisfy theoretical convergence properties . The magnitude of γ also plays a role in stabilizing learning dynamics of RL algorithms and has recently been treated as a hyperparameter of the optimization . However, both the magnitude and the functional form of this discounting function establish priors over the solutions learned. The magnitude of γ chosen establishes an effective horizon for the agent of 1/(1 − γ), far beyond which rewards are neglected . This effectively imposes a time-scale of the environment, which may not be accurate. Further, the exponential discounting of future rewards is consistent with a prior belief that there is a known constant per-time-step hazard rate or probability of dying of 1 − γ . Additionally, discounting future values exponentially and according to a single discount factor γ does not harmonize with the measured value preferences in humans 1 and animals (; ; ;). A wealth of empirical evidence has been amassed that humans, monkeys, rats and pigeons instead discount future returns hyperbolically, where d k (t) = 1 1+kt, for some positive k > 0 (; 1992; ; ; ;). This discrepancy between the time-preferences of animals from the exponential discounted measure of value might be presumed irrational. showed that hyperbolic time-preferences is mathematically consistent with the agent maintaining some uncertainty over the prior belief of the hazard rate in the environment. Hazard rate h(t) measures the per-time-step risk the agent incurs as it acts in the environment due to a potential early death. Precisely, if s(t) is the probability that the agent is alive at time t then the hazard rate is h(t) = − d dt lns(t). We consider the case where there is a fixed, but potentially unknown hazard rate h(t) = λ ≥ 0. The prior belief of the hazard rate p(λ) implies a specific discount function. Under this formalism, the canonical case in RL of discounting future rewards according to d(t) = γ t is consistent with the belief that there exists a single hazard rate λ = e −γ known with certainty. Further details are available in Appendix A. Figure 1: Hyperbolic versus exponential discounting. Humans and animals often exhibit hyperbolic discounts (blue curve) which have shallower discount declines for large horizons. In contrast, RL agents often optimize exponential discounts (orange curve) which drop at a constant rate regardless of how distant the return. Common RL environments are also characterized by risk, but often in a narrower sense. In deterministic environments like the original Arcade Learning Environment (ALE) stochasticity is often introduced through techniques like no-ops and sticky actions where the action execution is noisy. Physics simulators may have noise and the randomness of the policy itself induces risk. But even with these stochastic injections the risk to reward emerges in a more restricted sense. In Section 2 we show that a prior distribution reflecting the uncertainty over the hazard rate, has an associated discount function in the sense that an MDP with either this hazard distribution or the discount function, has the same value function for all policies. This equivalence implies that learning policies with a discount function can be interpreted as making them robust to the associated hazard distribution. Thus, discounting serves as a tool to ensure that policies deployed in the real world perform well even under risks they were not trained under. We propose an algorithm that approximates hyperbolic discounting while building on successful Qlearning tools and their associated theoretical guarantees. We show learning many Q-values, each discounting exponentially with a different discount factor γ, can be aggregated to approximate hyperbolic (and other non-exponential) discount factors. We demonstrate the efficacy of our approximation scheme in our proposed Pathworld environment which is characterized both by an uncertain per-time-step risk to the agent. Conceptually, Pathworld emulates a foraging environment where an agent must balance easily realizable, small meals versus more distant, fruitful meals. We then consider higher-dimensional deep RL agents in the ALE, where we measure the benefits of hyperbolic discounting. This approximation mirrors the work of; which empirically demonstrates that modeling a finite set of µAgents simultaneously can approximate hyperbolic discounting function. Our method then generalizes to other non-hyperbolic discount functions and uses deep neural networks to model the different Q-values from a shared representation. Surprisingly and in addition to enabling new non-exponential discounting schemes, we observe that learning a set of Q-values is beneficial as an auxiliary task . Adding this multi-horizon auxiliary task often improves over a state-of-the-art baseline, Rainbow in the ALE . This work questions the RL paradigm of learning policies through a single discount function which exponentially discounts future rewards through the following contributions: 1. Hazardous MDPs. We formulate MDPs with hazard present and demonstrate an equivalence between undiscounted values learned under hazards and (potentially nonexponentially) discounted values without hazard. 2. Hyperbolic (and other non-exponential)-agent. A practical approach for training an agent which discounts future rewards by a hyperbolic (or other non-exponential) discount function and acts according to this. 3. Multi-horizon auxiliary task. A demonstration of multi-horizon learning over many γ simultaneously as an effective auxiliary task. To study MDPs with hazard distributions and general discount functions we introduce two modifications. The hazardous MDP now is defined by the tuple < S, A, R, P, H, d >. In standard form, the state space S and the action space A may be discrete or continuous. The learner observes samples from the environment transition probability P (s t+1 |s t, a t) for going from s t ∈ S to s t+1 ∈ S given a t ∈ A. We will consider the case where P is a sub-stochastic transition function, which defines an episodic MDP. The environment emits a bounded reward r: S × A → [r min, r max] on each transition. In this work we consider non-infinite episodic MDPs. The first difference is that at the beginning of each episode, a hazard λ ∈ [0, ∞) is sampled from the hazard distribution H. This is equivalent to sampling a continuing probability γ = e −λ. During the episode, the hazard modified transition function will be P λ, in that P λ (s |s, a) = e −λ P (s |s, a). The second difference is that we now consider a general discount function d(t). This differs from the standard approach of exponential discounting in RL with γ according to d(t) = γ t, which is a special case. This setting makes a close connection to partially observable Markov Decision Process (POMDP) where one might consider λ as an unobserved variable. However, the classic POMDP definition contains an explicit discount function γ as part of its definition which does not appear here. A policy π: S → A is a mapping from states to actions. The state action value function Q is the expected discounted rewards after taking action a in state s and then following policy π until termination. where λ ∼ H and E π,P λ implies that s t+1 ∼ P λ (·|s t, a t) and a t ∼ π(·|s t). In the hazardous MDP setting we observe the same connections between hazard and discount functions delineated in Appendix A. This expresses an equivalence between the value function of an MDP with a discount and MDP with a hazard distribution. For example, there exists an equivalence between the exponential discount function d(t) = γ t to the undiscounted case where the agent is subject to a (1 − γ) per time-step of dying . The typical Q-value (left side of Equation 2) is when the agent acts in an environment without hazard λ = 0 or H = δ and discounts future rewards according to d(t) = γ t = e −λt which we denote as Q The alternative Q-value (right side of Equation 2) is when the agent acts under hazard rate λ = − ln γ but does not discount future rewards which we denote as Q where δ(x) denotes the Dirac delta distribution at x. This follows from P λ (s |s, a) = e −λ P (s |s, a) We also show a similar equivalence between hyperbolic discounting and the specific hazard distribu- For notational brevity later in the paper, we will omit the explicit hazard distribution H-superscript if the environment is not hazardous. This formulation builds upon's relate of hazard rate and discount functions and shows that this holds for generalized Q-values in reinforcement learning. We now show how one can re-purpose exponentially-discounted Q-values to compute hyperbolic (and other-non-exponential) discounted Q-values. The central challenge with using non-exponential discount strategies is that most RL algorithms use some form of TD learning . This family of algorithms exploits the Bellman equation which, when using exponential discounting, relates the value function at one state with the value at the following state. where expectation E π,P denotes sampling a ∼ π(·|s), s ∼ P (·|s, a), and a ∼ π(·|s). Being able to reuse TD methods without being constrained to exponential discounting is thus an important challenge. We propose here a scheme to deduce hyperbolic as well as other non-exponentially discounted Q-values when our discount function has a particular form. which we will refer to as the exponential weighting condition, then Proof. Applying the condition on d, The exchange in the above proof is valid if The exponential weighting condition is satisfied for hyperbolic discounting and other discounting that we might want to consider (see Appendix F for examples). As an example, the hyperbolic discount can be expressed as the integral of a function f (γ, t) for γ = in Equation 9. This equationn tells us an integral over a function f (γ, t) = 1 k γ 1/k+t−1 = w(γ)γ t yields the desired hyperbolic discount factor Γ k (t) = This prescription gives us a tool to produce general forms of non-exponentially discounted Q-values using our familiar exponentially discounted Q-values traditionally learned in RL . Section 3 describes an equivalence between hyperbolically-discounted Q-values and integrals of exponentially-discounted Q-values, however, the method required evaluating an infinite set of value functions. We therefore present a practical approach to approximate discounting Γ(t) = 1 1+kt using a finite set of functions learned via standard Q-learning . To avoid estimating an infinite number of Q γ π -values we introduce a free hyperparameter (n γ) which is the total number of Q γ π -values to consider, each with their own γ. We use a practically-minded approach to choose G that emphasizes evaluating larger values of γ rather than uniformly choosing points and empirically performs well as seen in Section 5. Our approach is described in Appendix G. Each Q γi π computes the discounted sum of returns according to that specific discount factor We previously proposed two equivalent approaches for computing hyperbolic Q-values, but for simplicity we consider the one presented in Lemma 3.1. The set of Q-values permits us to estimate the integral through a Riemann sum (Equation 11) which is described in further detail in Appendix I. where we estimate the integral through a lower bound. We consolidate this entire process in Figure 11 where we show the full process of rewriting the hyperbolic discount rate, hyperbolically-discounted Q-value, the approximation and the instantiated agent. This approach is similar to that of where each µAgent models a specific discount factor γ. However, this differs in that our final agent computes a weighted average over each Q-value rather than a sampling operation of each agent based on a γ-distribution. The benefits of hyperbolic discounting will be greatest under two conditions: uncertain hazard and non-trivial intertemporal decisions. The first condition can arise under a unobserved hazard-rate variable λ drawn independently at the beginning of each episode from H = p(λ). The second condition emerges with a choice between a smaller nearby rewards versus larger distant rewards. In the absence of both properties we would not expect any advantage to discounting hyperbolically. To see why, if there is a single-true hazard rate λ env, than an optimal γ * = e −λenv exists and future rewards should be discounted exponentially according to it. Further, if there is a single path through the environment with perfect alignment of short-and long-term objectives, all discounting schemes yield the same optimal policy. We note two sources for discounting rewards in the future: time delay and survival probability (Section 2). In Pathworld we train to maximize hyperbolically discounted returns (t Γ k (t)R(s t, a t)) under no hazard (H = δ(λ − 0)) but then evaluate the undiscounted returns d(t) = 1.0 ∀ t with the paths subject to hazard H = 1 k exp(−λ/k). Through this procedure, we are able to train an agent that is robust to hazards in the environment. The agent makes one decision in Pathworld (Figure 2): which of the N paths to investigate. Once a path is chosen, the agent continues until it reaches the end or until it dies. This is similar to a multi-armed bandit, with each action subject to dynamic risk. The paths vary quadratically in length with the index d(i) = i 2 but the rewards increase linearly with the path index r(i) = i. This presents... Figure 2: The Pathworld. Each state (white circle) indicates the accompanying reward r and the distance from the starting state d. From the start state, the agent makes a single action: which which path to follow to the end. Longer paths have a larger rewards at the end, but the agent incurs a higher risk on a longer path. a non-trivial decision for the agent. At deployment, an unobserved hazard λ ∼ H is drawn and the agent is subject to a per-time-step risk of dying of (1 − e −λ). This environment differs from the adjusting-delay procedure presented by and then later modified by. Rather then determining time-preferences through variable-timing of rewards, we determine time-preferences through risk to the reward. Figure 3: In each episode of Pathworld an unobserved hazard λ ∼ p(λ) is drawn and the agent is subject to a total risk of the reward not being realized of Figure 3 showing that our approximation scheme well-approximates the true valueprofile. Figure 3 validates that our approach well-approximates the true hyperbolic value of each path when the hazard prior matches the true distribution. Agents that discount exponentially according to a single γ (the typical case in RL) incorrectly value the paths. We examine further the failure of exponential discounting in this hazardous setting. For this environment, the true hazard parameter in the prior was k = 0.05 (i.e. λ ∼ 20exp(−λ/0.05)). Therefore, at deployment, the agent must deal with dynamic levels of risk and faces a non-trivial decision of which path to follow. Even if we tune an agent's γ = 0.975 such that it chooses the correct arg-max path, it still fails to capture the functional form (Figure 3) and it achieves a high error over all paths (Table 1). If the arg-max action was not available or if the agent was proposed to evaluate non-trivial intertemporal decisions, it would act sub-optimally. In Appendix B we consider additional experiments where the agent's prior over hazard more realistically does not exactly match the environment true hazard rate and demonstrate the benefit of appropriate priors. With our approach validated in Pathworld, we now move to the high-dimensional environment of Atari 2600, specifically, ALE. We use the Rainbow variant from Dopamine which implements three of the six considered improvements from the original paper: distributional RL, predicting n-step returns and prioritized replay buffers. The agent (Figure 4) maintains a shared representation h(s) of state, but computes Q-value logits for each of the N γ i via Q π (s, a) = W i h(s) + b i where W i and b i are the learnable parameters of the affine transformation for that head. A ReLU-nonlinearity is used within the body of the network .: Multi-horizon model predicts Q-values for n γ separate discount functions thereby modeling different effective horizons. Each Q-value is a lightweight computation, an affine transformation off a shared representation. By modeling over multiple time-horizons, we now have the option to construct policies that act according to a particular value or a weighted combination. Hyperparameter details are provided in Appendix K and when applicable, they default to the standard Dopamine values. We find strong performance improvements of the hyperbolic agent built on Rainbow (Hyper-Rainbow; blue bars) on a random subset of Atari 2600 games in Figure 5. To dissect the Hyper-Rainbow improvements, recognize that two properties from the base Rainbow agent have changed: 1. Behavior policy, µ. The agent acts according to hyperbolic Q-values computed by our approximation described in Section 4 2. Learn over multiple horizons. The agent simultaneously learns Q-values over many γ rather than a Q-value for a single γ On this subset of 19 games, Hyper-Rainbow improves upon 14 games and in some cases, by large margins. But we seek here a more complete understanding of the underlying driver of this improvement in ALE through an ablation study. The second modification can be regarded as introducing an auxiliary task . Therefore, to attribute the performance of each properly we construct a Rainbow agent augmented with the multi-horizon auxiliary task (referred to as Multi-Rainbow and shown in orange) but have it still act according to the original policy. That is, Multi-Rainbow acts to maximize expected rewards discounted by a fixed γ action but now learns over multiple horizons as shown in Figure 4. Figure 5: We compare the Hyper-Rainbow (in blue) agent versus the Multi-Rainbow (orange) agent on a random subset of 19 games from ALE (3 seeds each). For each game, the percentage performance improvement for each algorithm against Rainbow is recorded. There is no significant difference whether the agent acts according to hyperbolically-discounted (Hyper-Rainbow) or exponentiallydiscounted (Multi-Rainbow) Q-values suggesting the performance improvement in ALE emerges from the multi-horizon auxiliary task. We find that the Multi-Rainbow agent performs nearly as well on these games, suggesting the effectiveness of this as a stand-alone auxiliary task. This is not entirely unexpected given the rather special-case of hazard exhibited in ALE through sticky-actions . We examine further and investigate the performance of this auxiliary task across the full Arcade Learning Environment using the recommended evaluation by . Doing so we find strong empirical benefits of the multi-horizon auxiliary task over the state-of-the-art Rainbow agent as shown in Figure 6. Game Name Auxiliary Task Improvement for Rainbow Agent Figure 6: Performance improvement over Rainbow using the multi-horizon auxiliary task in Atari Learning Environment (3 seeds each). To understand the interplay of the multi-horizon auxiliary task with other improvements in deep RL, we test a random subset of 10 Atari 2600 games against improvements in Rainbow . On this set of games we measure a consistent improvement with multi-horizon C51 (Multi-C51) in 9 out of the 10 games over the base C51 agent in Figure 7. Figure 7 indicates that the current implementation of Multi-Rainbow does not generally build successfully on the prioritized replay buffer. On the subset of ten games considered, we find that four out of ten games (Pong, Venture, Gravitar and Zaxxon) are negatively impacted despite finding it to be of considerable benefit and specifically beneficial in three out of these Hyperbolic discounting in economics. Hyperbolic discounting is well-studied in the field of economics . proposes a softer interpretation than (which produces a per-time-step of death via the hazard rate) and demonstrates that uncertainty over the timing of rewards can also give rise to hyperbolic discounting and preference reversals, a hallmark of hyperbolic discounting. Hyperbolic discounting was initially presumed to not lend itself to TD-based solutions but the field has evolved on this point. proposes solution directions that find models that discount quasi-hyperbolically even though each learns with exponential discounting but reaffirms the difficulty. proposes hyperbolically discounted temporal difference (HDTD) learning by making connections to hazard. Behavior RL and hyperbolic discounting in neuroscience. TD-learning has long been used for modeling behavioral reinforcement learning (; ;). TD-learning computes the error as the difference between the expected value and actual value where the error signal emerges from unexpected rewards. However, these computations traditionally rely on exponential discounting as part of the estimate of the value which disagrees with empirical evidence in humans and animals (; ; ; 1992). Hyperbolic discounting has been proposed as an alternative to exponential discounting though it has been debated as an accurate model . Naive modifications to TD-learning to discount hyperbolically present issues since the demonstrated that distributed exponential discount factors can directly model hyperbolic discounting. This work proposes the µAgent, an agent that models the value function with a specific discount factor γ. When the distributed set of µAgent's votes on the action, this was shown to approximate hyperbolic discounting well in the adjusting-delay assay experiments . Using the hazard formulation established in , we demonstrate how to extend this to other non-hyperbolic discount functions and demonstrate the efficacy of using a deep neural network to model the different Q-values from a shared representation. Towards more flexible discounting in reinforcement learning. RL researchers have recently adopted more flexible versions beyond a fixed discount factor (; ; ;). Optimal policies are studied in where two value functions with different discount factors are used. Introducing the discount factor as an argument to be queried for a set of timescales is considered in both Horde and γ-nets . proposes the Average Reward Independent Gamma Ensemble framework which imitates the average return estimator. generalizes the original discounting model through discount functions that vary with the age of the agent, expressing time-inconsistent preferences as in hyperbolic discounting. The need to increase training stability via effective horizon was addressed in François- who proposed dynamic strategies for the discount factor γ. Meta-learning approaches to deal with the discount factor have been proposed in Xu, van. characterizes rational decision making in sequential processes, formalizing a process that admits a state-action dependent discount rates. Operating over multiple time scales has a long history in RL. generalizes the work of and to formalize a multi-time scale TD learning model theory. Previous work has been explored on solving MDPs with multiple reward functions and multiple discount factors though these relied on separate transition models . considers decomposing a reward function into separate components each with its own discount factor. In our work, we continue to model the same rewards, but now model the value over different horizons. Recent work in difficult exploration games demonstrates the efficacy of two different discount factors one for intrinsic rewards and one for extrinsic rewards. Finally, and concurrent with this work, proposes the TD(∆)-algorithm which breaks a value function into a series of value functions with smaller discount factors. Auxiliary tasks in reinforcement learning. Finally, auxiliary tasks have been successfully employed and found to be of considerable benefit in RL. used auxiliary tasks to facilitate representation learning. Building upon this, work in RL has consistently demonstrated benefits of auxiliary tasks to augment the low-information coming from the environment through extrinsic rewards (; ; ; ;) 8 DISCUSSION AND FUTURE WORK This work builds on a body of work that questions one of the basic premises of RL: one should maximize the exponentially discounted returns via a single discount factor. By learning over multiple horizons simultaneously, we have broadened the scope of our learning algorithms. Through this we have shown that we can enable acting according to new discounting schemes and that learning multiple horizons is a powerful stand-alone auxiliary task. Our method well-approximates hyperbolic discounting and performs better in hazardous MDP distributions. This may be viewed as part of an algorithmic toolkit to model alternative discount functions. However, this work still does not fully capture more general aspects of risk since the hazard rate may be a function of time. Further, hazard may not be an intrinsic property of the environment but a joint property of both the policy and the environment. If an agent purses a policy leading to dangerous state distributions then it will naturally be subject to higher hazards and vice-versathis creates a complicated circular dependency. We would therefore expect an interplay between time-preferences and policy. This is not simple to deal with but recent work proposing state-action dependent discounting formalizes time preferences in which future rewards are discounted based on the probability that the agent will not survive to collect them due to an encountered risk or hazard. Definition A.1. Survival s(t) is the probability of the agent surviving until time t. s(t) = P (agent is alive|at time t) A future reward r t is less valuable presently if the agent is unlikely to survive to collect it. If the agent is risk-neutral, the present value of a future reward r t received at time-t should be discounted by the probability that the agent will survive until time t to collect it, s(t). Consequently, if the agent is certain to survive, s(t) = 1, then the reward is not discounted per Equation 14. From this it is then convenient to define the hazard rate. Definition A.2. Hazard rate h(t) is the negative rate of change of the log-survival at time t or equivalently expressed as h(t) = − ds(t) dt 1 s(t). Therefore the environment is considered hazardous at time t if the log survival is decreasing sharply. demonstrates that the prior belief of the risk in the environment implies a specific discounting function. When the risk occurs at a known constant rate than the agent should discount future rewards exponentially. However, when the agent holds uncertainty over the hazard rate then hyperbolic and alternative discounting rates arise. We recover the familiar exponential discount function in RL based on a prior assumption that the environment has a known constant hazard. Consider a known hazard rate of h(t) = λ ≥ 0. Definition A.2 sets a first order differential equation. The solution for the survival rate is s(t) = e −λt which can be related to the RL discount factor γ This interprets γ as the per-time-step probability of the episode continuing. This also allows us to connect the hazard rate λ ∈ [0, ∞] to the discount factor γ ∈. As the hazard increases λ → ∞, then the corresponding discount factor becomes increasingly myopic γ → 0. Conversely, as the environment hazard vanishes, λ → 0, the corresponding agent becomes increasingly far-sighted γ → 1. In RL we commonly choose a single γ which is consistent with the prior belief that there exists a known constant hazard rate λ = −ln(γ). We now relax the assumption that the agent holds this strong prior that it exactly knows the true hazard rate. From a Bayesian perspective, a looser prior allows for some uncertainty in the underlying hazard rate of the environment which we will see in the following section. We may not always be so confident of the true risk in the environment and instead reflect this underlying uncertainty in the hazard rate through a hazard prior p(λ). Our survival rate is then computed by weighting specific exponential survival rates defined by a given λ over our prior p(λ) shows that under an exponential prior of hazard p(λ) = 1 k exp(−λ/k) the expected survival rate for the agent is hyperbolic We denote the hyperbolic discount by Γ k (t) to make the connection to γ in reinforcement learning explicit. shows that different priors over hazard correspond to different discount functions. We reproduce two figures in Figure 8 showing the correspondence between different hazard rate priors and the ant discount functions. The common approach in RL is to maintain a delta-hazard (black line) which leads to exponential discounting of future rewards. Different priors lead to non-exponential discount functions. There is a correspondence between hazard rate priors and the ing discount function. In RL, we typically discount future rewards exponentially which is consistent with a Dirac delta prior (black line) on the hazard rate indicating no uncertainty of hazard rate. However, this is a special case and priors with uncertainty over the hazard rate imply new discount functions. All priors have the same mean hazard rate E[p(λ)] = 1. In Figure 9 we consider the case that the agent still holds an exponential prior but has the wrong coefficient k and in Figure 10 we consider the case where the agent still holds an exponential prior but the true hazard is actually drawn from a uniform distribution with the same mean. Through these two validating experiments, we demonstrate the robustness of estimating hyperbolic discounted Q-values in the case when the environment presents dynamic levels of risk and the agent faces non-trivial decisions. Hyperbolic discounting is preferable to exponential discounting even when the agent's prior does not precisely match the true environment hazard rate distribution, by coefficient (Figure 9) or by functional form (Figure 10).. Predictably, the mismatched priors in a higher prediction error of value but performs more reliably than exponential discounting, ing in a cumulative lower error. Numerical in Table 2. Table 2: The average mean squared error (MSE) over each of the paths in Figure 9. As the prior is further away from the true value of k = 0.05, the error increases. However, notice that the errors for large factor-of-2 changes in k in generally lower errors than if the agent had considered only a single exponential discount factor γ as in Table 1. Figure 10: If the true hazard rate is now drawn according to a uniform distribution (with the same mean as before) the original hyperbolic discount matches the functional form better than exponential discounting. Numerical in Table 3. hyperbolic value 0.235 γ = 0.975 0.266 γ = 0.95 0.470 γ = 0.99 4.029 Table 3: The average mean squared error (MSE) over each of the paths in Figure 10 when the underlying hazard is drawn according to a uniform distribution. We find that hyperbolic discounting is more robust to hazards drawn from a uniform distribution than exponential discounting. Let's start with the case where we would like to estimate the value function where rewards are discounted hyperbolically instead of the common exponential scheme. We refer to the hyperbolic Q-values as Q Γ π below in Equation 21 We may relate the hyperbolic Q Γ π -value to the values learned through standard Q-learning. To do so, notice that the hyperbolic discount Γ t can be expressed as the integral of a certain function f (γ, t) for γ = in Equation 22. The integral over this specific function f (γ, t) = γ kt yields the desired hyperbolic discount factor Γ k (t) by considering an infinite set of exponential discount factors γ over its domain γ ∈. Recognize that the integrand γ kt is the standard exponential discount factor which suggests a connection to standard Q-learning . This suggests that if we could consider an infinite set of γ then we can combine them to yield hyperbolic discounts for the corresponding time-step t. We build on this idea of modeling many γ throughout this work. where Γ k (t) has been replaced on the first line by as a weighting over exponentiallydiscounted Qvalues using the same weights: can be expressed as a weighting over exponential discount functions with weights (see Table 1). 3. The integral in box 2 can be approximated with a Riemann sum over the discrete intervals: Following Section A we also show a similar equivalence between hyperbolic discounting and the specific hazard distribution Where the first step uses Equation 19. This equivalence implies that discount factors can be used to learn policies that are robust to hazards. We expand upon three special cases to see how functions f (γ, t) = w(γ)γ t may be related to different discount functions d(t). We summarize in Table 4 how a particular hazard prior p(λ) can be computed via integrating over specific weightings w(γ) and the corresponding discount function. Table 4: Different hazard priors H = p(λ) can be alternatively expressed through weighting exponential discount functions γ t by w(γ). This table matches different hazard distributions to their associated discounting function and the weighting function per Lemma 3.1. The typical case in RL is a Dirac Delta Prior over hazard rate δ(λ − k). We only show this in detail for completeness; one would not follow such a convoluted path to arrive back at an exponential discount but this approach holds for richer priors. The derivations can be found in the Appendix F. Three cases: For the three cases we begin with the Laplace transform on the prior p(λ) = ∞ λ=0 p(λ)e −λt dλ and then chnage the variables according to the relation between γ = e −λ, Equation 17. A delta prior p(λ) = δ(λ − k) on the hazard rate is consistent with exponential discounting. where δ(λ − k) is a Dirac delta function defined over variable λ with value k. The change of variable γ = e −λ (equivalently λ = − ln γ) yields differentials dλ = − 1 γ dγ and the limits λ = 0 → γ = 1 and λ = ∞ → γ = 0. Additionally, the hazard rate value λ = k is equivalent to the γ = e −k. where we define a γ k = e −k to make the connection to standard RL discounting explicit. Additionally and reiterating, the use of a single discount factor, in this case γ k, is equivalent to the prior that a single hazard exists in the environment. Again, the change of variable γ = e −λ yields differentials dλ = − 1 γ dγ and the limits λ = 0 → γ = 1 and λ = ∞ → γ = 0. where p(·) is the prior. With the exponential prior p(λ) = 1 k exp(−λ/k) and by substituting λ = −lnγ we verify Equation 9 Finally if we hold a uniform prior over hazard, shows the Laplace transform yields Use the same change of variables to relate this to γ. The bounds of the integral become λ = 0 → γ = 1 and λ = k → γ = e −k. which recovers the discounting scheme. We provide further detail for which γ we choose to model and motivation why. We choose a γ max which is the largest γ to learn through Bellman updates. If we are using k as the hyperbolic coefficient in Equation 19 and we are approximating the integral with n γ our γ max would be However, allowing γ max → 1 get arbitrarily close to 1 may in learning instabilities. Therefore we compute an exponentiation base of b = exp(ln(1 − γ 1/k max)/n γ ) which bounds our γ max at a known stable value. This induces an approximation error which is described more in Appendix H. Instead of evaluating the upper bound of Equation 9 at 1 we evaluate at γ max which yields γ kt max /(1+kt). Our approximation induces an error in the approximation of the hyperbolic discount. This approximation error in the Riemann sum increases as the γ max decreases as evidenced by Figure 12. When the maximum value of γ max → 1 then the approximation becomes more accurate as supported in Table 5 up to small random errors. As discussed, we can estimate the hyperbolic discount in two different ways. We illustrate the ing estimates here and ing approximations. We use lower-bound Riemann sums in both cases for simplicity but more sophisticated integral estimates exist. As noted earlier, we considered two different integrals for computed the hyperbolic coefficients. Under the form derived by the Laplace transform, the integrals are sharply peaked as γ → 1. The difference in integrals is visually apparent comparing in Figure 13. Figure 12: By instead evaluating our integral up to γ max rather than to 1, we induce an approximation error which increases with t. Numerical in Table 5. Figure 12. (a) We approximate the integral of the function γ kt via a lower estimate of rectangles at specific γ-values. The sum of these rectangles approximates the hyperbolic discounting scheme 1/(1 + kt) for time t. (b) Alternative form for approximating hyperbolic coefficients which is sharply peaked as γ → 1 which led to larger errors in estimation under our initial techniques. J PERFORMANCE OF DIFFERENT REPLAY BUFFER PRIORITIZATION SCHEME As found through our ablation study in Figure 7, the Multi-Rainbow auxiliary task interacted poorly with the prioritized replay buffer when the TD-errors were averaged evenly across all heads. As an alternative scheme, we considered prioritizing according to the largest γ, which is also the γ defining the Q-values by which the agent acts. The (preliminary 5) of this new prioritization scheme is in Figure 14. -10 3 -10 2 -10 1 -10 0 0 10 0 10 1 10 2 10 3 Percent Improvement (log) Game Name Multi-Rainbow Improvement over Rainbow (prioritize-largest) Figure 14: The (preliminary) performance improvement over Rainbow using the multi-horizon auxiliary task in Atari Learning Environment when we instead prioritize according to the TD-errors computed from the largest γ (3 seeds each). To this point, there is evidence that prioritizing according to the TD-errors generated by the largest gamma is a better strategy than averaging. Final of the multi-horizon auxiliary task on Rainbow (Multi-Rainbow) in Table 7.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

A deep RL agent that learns hyperbolic (and other non-exponential) Q-values and a new multi-horizon auxiliary task.

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.


Question: Represent the article for finding a single-sentence TLDR

Question: Emotion is playing a great role in our daily lives. The necessity and importance of an automatic Emotion recognition system is getting increased. Traditional approaches of emotion recognition are based on facial images, measurements of heart rates, blood pressure, temperatures, tones of voice/speech, etc. However, these features can potentially be changed to fake features. So to detect hidden and real features that is not controlled by the person are data measured from brain signals. There are various ways of measuring brain waves: EEG, MEG, FMRI, etc. On the bases of cost effectiveness and performance trade-offs, EEG is chosen for emotion recognition in this work. The main aim of this study is to detect emotion based on EEG signal analysis recorded from brain in response to visual stimuli. The approaches used were the selected visual stimuli were presented to 11 healthy target subjects and EEG signal were recorded in controlled situation to minimize artefacts (muscle or/and eye movements). The signals were filtered and type of frequency band was computed and detected. The proposed method predicts an emotion type (positive/negative) in response to the presented stimuli. Finally, the performance of the proposed approach was tested. The average accuracy of machine learning algorithms (i.e. J48, Bayes Net, Adaboost and Random Forest) are 78.86, 74.76, 77.82 and 82.46 respectively. In this study, we also applied EEG applications in the context of neuro-marketing. The empirically demonstrated detection of the favourite colour preference of customers in response to the logo colour of an organization or Service. Emotion is playing a great role in our daily lives. The necessity and importance of an automatic 25 Emotion recognition system is getting increased. Traditional approaches of emotion recognition 26 are based on facial images, measurements of heart rates, blood pressure, temperatures, tones of 27 voice/speech, etc. However, these features can potentially be changed to fake features. Thus, to simple and portable device. The brainwave activity is broadly divided into five frequency bands. The boundary between the frequency bands is not strict but not varying much. The frequency bands 63 include delta(0.5-4Hz), theta(5-8Hz), alpha(9-12Hz), beta(13-30Hz) and gamma(above 30Hz). For this study, EEG data is collected using Emotiv EPOC device with 14 electrodes located at AF3, gamma band is also responsible for arousal. In other words, the EEG brain activity from parietal 90 and frontal lobe of the brain is more emotionally informative where as gamma, alpha and beta waves We collected and prepared three image data sets for stimuli presentation and classification. These To answer research question 1 and 2, the top ranked features for each subject are extracted using

Represent the TLDR for finding the corresponding scientific article.

This paper presents EEG based emotion detection of a person towards an image stimuli and its applicability on neuromarketing.

we present the state-of-the-art results of using neural networks to diagnose chest x-rays


Question: Represent the article for finding a single-sentence TLDR.

Question: We present a probabilistic framework for session based recommendation. A latent variable for the user state is updated as the user views more items and we learn more about their interests. We provide computational solutions using both the re-parameterization trick and using the Bouchard bound for the softmax function, we further explore employing a variational auto-encoder and a variational Expectation-Maximization algorithm for tightening the variational bound. Finally we show that the Bouchard bound causes the denominator of the softmax to decompose into a sum enabling fast noisy gradients of the bound giving a fully probabilistic algorithm reminiscent of word2vec and a fast online EM algorithm. Our model describes a generative process for the types of products that user's co-view in sessions. We use u to denote a user or a session, we use t time to denote sequential time and v to denote which product they viewed from 1 to P where P is the number of products, the user's interest is described by a K dimensional latent variable ω u which can be interpreted as the user's interest in K topics. The session length of user u is given by T u. We then assume the following generative process for the views in each session: ω u ∼ N (0 K, I K), v u,1,.., v u,Tu ∼ categorical(softmax(Ψω u + ρ)). This model is a linear version of the model presented in. Consider the case where we have estimated that Ψ and ρ. In production we have observed a user's online viewing history v u,1,.., v u,Tu and we would like to produce a representation of the user's interests. Our proposal is to use Bayesian inference in order to infer p(ω|v u,1, .., v u,Tu, Ψ, ρ) as a representation of their interests. This representation of interests can then be used as a feature for training a recommender system. If we have a recommender system that has just seven products and the products have embeddings: We now consider how different user histories affect p(ω|v u,1, .., v u,Tu, Ψ, ρ). Approximation of this quantity can be made accurately and easily using the Stan probabilistic programming language or using variational approximations. In Figure 1 -2 the intuitive behavior of this simple model is demonstrated. The of the three approximate methods are presented and shown to be in good agreement. A single product view indicates interest in that class of products, but considerable uncertainty remains. Many product views in the same class represent high certainty that the user is interested in that class. For next item prediction we consider both taking the plug-in predictive based on the posterior mean VB (approx) and using Monte Carlo samples to approximate the true predictive distribution MCMC and VB (MC). The model we introduce has the form: If we use a normal distribution ω ∼ N (µ q, Σ q), then variational bound has the form: but we still need to be able to integrate under the denominator of the softmax. The Bouchard bound introduces a further approximation and additional variational parameters a, ξ but produces an analytical bound: Where λ JJ (·) is the Jaakola and Jordan function:. Alternatively the re-parameterization trick proceeds by simulating: (s) ∼ N (0 K, I K), and then computing:, and then optimizing the noisy lower bound: In order to prevent the variational parameters growing with the amount of data we employ a variational auto-encoder. This involves using a flexible function i.e. µ q, Σ q = f Ξ (v 1, ...v T), or in the case of the Bouchard bound: Where any function (e.g. a deep net) can be used for f Ξ (·) and f Bouch Ξ (·). We demonstrate that our method using the RecoGym simulation environment. We fit the model to the training set, we then evaluate by providing the model v 1,..v Tu−1 events and testing the model's ability to predict v Tu. A further approximate algorithm which is useful when P is large is to note that the bound can be written as a sum that decomposes not only in data but also over the denominator of the softmax, The noisy lower bound becomes: where v 1,..., v T are the items associated with the session and n 1,...n S are S < P negative items randomly sampled. Similar to the word2vec algorithm but without any non-probabilistic heuristics. We consider two alternative methods for training the model: Bouch/AE -A linear variational auto-encoder using the Bouchard bound; RT/Deep AE -A deep auto-encoder again using the re-parameterization trick. The deep auto-encoder consists of mapping an input of size P to three linear rectifier layers of K units each. Results showing recall@5 and discounted cumulative gain at 5 are shown in Table 1. The EM algorithm allows an approximation to be made of q(ω) assuming (Ψ, ρ) and a user history v 1,.., v T are known and can be used in place of a variational auto-encoder. The algorithm here is the dual of the one presented in as we assume the embedding Ψ is fixed and ω is updated where the algorithm they present does the opposite. The EM algorithm consists of cycling the following update equations: We further note that the EM algorithm is (with the exception of the a variational parameter) a fixed point update (of the natural parameters) that decomposes into a sum. The terms in the sum come from the softmax in the denominator. After substituting a co-ordinate descent update of a with a gradient descent step update, then the entire fixed point update becomes a sum: That is the EM algorithm can be written: As noted in Cappé and when an EM algorithm can be written as a fixed point update over a sum, then the Robbins Monro algorithm can be applied. Allowing updates of the form (p is chosen randomly): where ∆ is a slowly decaying Robbins Monro sequence with ∆ 1 = 1 (meaning no initial value of (Σ −1) ) is needed. For large P this algorithm is many times faster than the generic EM algorithm. What is distinct about both this online EM algorithm and the negative sampling SGD approach is that it is the denominator of the softmax that may be sub-sampled rather than individual records. The Bouchard bound is also used for decomposing the softmax into a sum in but they do per-batch optimization of the variational parameters, instead we use an auto-encoder allowing direct SGD. Our method also differs from again in using an auto-encoder allowing the more flexible SGD algorithm in place of stochastic variational inference which requires complete data exponential family assumptions. Finally unlike those methods we are considering variational inference of a latent variable model as well as using variational bounds to approximate the softmax.

Fast variational approximations for approximating a user state and learning product embeddings

Represent the short summary for finding the corresponding scientific article (long)

Propose first methods for exactly optimizing the softmax distribution using stochastic gradient with runtime independent on the number of classes or datapoints.