Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (TD3), another off-policy deep RL algorithm which improves over DDPG. We evaluate the ing method, CEM-RL, on a set of benchmarks classically used in deep RL. We show that CEM-RL benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency. Policy search is the problem of finding a policy or controller maximizing some unknown utility function. Recently, research on policy search methods has witnessed a surge of interest due to the combination with deep neural networks, making it possible to find good enough continuous action policies in large domains. From one side, this combination gave rise to the emergence of efficient deep reinforcement learning (deep RL) techniques BID17 BID25. From the other side, evolutionary methods, and particularly deep neuroevolution methods applying evolution strategies (ESs) to the parameters of a deep network have emerged as a competitive alternative to deep RL due to their higher parallelization capability BID23.Both families of techniques have clear distinguishing properties. Evolutionary methods are significantly less sample efficient than deep RL methods because they learn from complete episodes, whereas deep RL methods use elementary steps of the system as samples, and thus exploit more information BID27. In particular, off-policy deep RL algorithms can use a replay buffer to exploit the same samples as many times as useful, greatly improving sample efficiency. Actually, the sample efficiency of ESs can be improved using the "importance mixing" mechanism, but a recent study has shown that the capacity of importance mixing to improve sample efficiency by a factor of ten is still not enough to compete with off-policy deep RL BID22. From the other side, sample efficient off-policy deep RL methods such as the DDPG algorithm BID17 are known to be unstable and highly sensitive to hyper-parameter setting. Rather than opposing both families as competing solutions to the policy search problem, a richer perspective consists in combining them so as to get the best of both worlds. As covered in Section 2, there are very few attempts in this direction so far. After presenting some in Section 3, we propose in Section 4 a new combination method that combines the cross-entropy method (CEM) with DDPG or TD3, an off-policy deep RL algorithm which improves over DDPG. In Section 5, we investigate experimentally the properties of this CEM-RL method, showing its advantages both over the components taken separately and over a competing approach. Beyond the of CEM-RL, the of this work is that there is still a lot of unexplored potential in new combinations of evolutionary and deep RL methods. Policy search is an extremely active research domain. The realization that evolutionary methods are an alternative to continuous action reinforcement learning and that both families share some similarity is not new BID29 b; 2013) but so far most works have focused on comparing them BID24. Under this perspective, it was shown in BID6 that, despite its simplicity with respect to most deep RL methods, the Cross-Entropy Method (CEM) was a strong baseline in policy search problems. Here, we focus on works which combine both families of methods. Synergies between evolution and reinforcement learning have already been investigated in the context of the so-called Baldwin effect BID28. This literature is somewhat related to research on meta-learning, where one seeks to evolve an initial policy from which a self-learned reinforcement learning algorithm will perform efficient improvement (; BID12 BID9 . The key difference with respect to the methods investigated here is that in this literature, the outcome of the RL process is not incorporated back into the genome of the agent, whereas here evolution and reinforcement learning update the same parameters in iterative sequences. Closer to ours, the work of BID3 sequentially applies a goal exploration process (GEP) to fill a replay buffer with purely exploratory trajectories and then applies DDPG to the ing data. The GEP shares many similarities with evolutionary methods, though it focuses on diversity rather than on performance of the learned policies. The authors demonstrate on the Continuous Mountain Car and HALF-CHEETAH-V2 benchmarks that their combination, GEP-PG, is more sample-efficient than DDPG, leads to better final solutions and induces less variance during learning. However, due to the sequential nature of the combination, the GEP part does not benefit from the efficient gradient steps of the deep RL part. Another approach related to ours is the work of BID18, where the authors introduce optimization problems with a surrogate gradient, i.e. a direction which is correlated with the real gradient. They show that by modifying the covariance matrix of an ES to incorporate the informations contained in the surrogate, a hybrid algorithm can be constructed. They provide a thorough theoretical investigation of their procedure, which they experimentally show capable of outperforming both a standard gradient descent method and a pure ES on several simple benchmarks. They argue that this method could be useful in RL, since surrogate gradients appear in Q-learning and actor-critic methods. However, a practical demonstration of those claims remains to be performed. Their approach resembles ours in that they use a gradient method to enhance an ES. But a notable difference is that they use the gradient information to directly change the distribution from which samples are drawn, whereas we use gradient information on the samples themselves, impacting the distribution only indirectly. The work which is the closest to ours is BID14. The authors introduce an algorithm called ERL (for Evolutionary Reinforcement Learning), which is presented as an efficient combination of a deep RL algorithm, DDPG, and a population-based evolutionary algorithm. It takes the form of a population of actors, which are constantly mutated and selected in tournaments based on their fitness. In parallel, a single DDPG agent is trained from the samples generated by the evolutionary population. This single agent is then periodically inserted into the population. When the gradient-based policy improvement mechanism of DDPG is efficient, this individual outperforms its evolutionary siblings, it gets selected into the next generation and draws the whole population towards higher performance. Through their experiments, Khadka & Tumer demonstrate that this setup benefits from an efficient transfer of information between the RL algorithm and the evolutionary algorithm, and vice versa. However, their combination scheme does not make profit of the search efficiency of ESs. This is unfortunate because ESs are generally efficient evolutionary methods, and importance mixing can only be applied in their context to bring further sample efficiency improvement. By contrast with the works outlined above, the method presented here combines CEM and TD3 in such a way that our algorithm benefits from the gradient-based policy improvement mechanism of TD3, from the better stability of ESs, and may even benefit from the better sample efficiency brought by importance mixing, as described in Appendix B. In this section, we provide a quick overview of the evolutionary and deep RL methods used throughout the paper. Evolutionary algorithms manage a limited population of individuals, and generate new individuals randomly in the vicinity of the previous elite individuals. There are many variants of such algorithms, some using tournament selection as in BID14, niche-based selection or more simply taking a fraction of elite individuals, see BID1 for a broader view. Evolution strategies can be seen as specific evolutionary algorithms where only one individual is retained from one generation to the next, this individual being the mean of the distribution from which new individuals are drawn. More specifically, an optimum individual is computed from the previous samples and the next samples are obtained by adding Gaussian noise to the current optimum. Finally, among ESs, Estimation of Distribution Algorithms (EDAs) are a specific family where the population is represented as a distribution using a covariance matrix Σ BID16 ). This covariance matrix defines a multivariate Gaussian function and samples at the next iteration are drawn according to Σ. Along iterations, the ellipsoid defined by Σ is progressively adjusted to the top part of the hill corresponding to the local optimum θ *. Various instances of EDAs, such as the CrossEntropy Method (CEM), Covariance Matrix Adaptation Evolution Strategy (CMA-ES) and PI 2 -CMA, are covered in BID29 b; 2013). Here we focus on the first two. The Cross-Entropy Method (CEM) is a simple EDA where the number of elite individuals is fixed to a certain value K e (usually set to half the population). After all individuals of a population are evaluated, the K e fittest individuals are used to compute the new mean and variance of the population, from which the next generation is sampled after adding some extra variance to prevent premature convergence. In more details, each individual x i is sampled by adding Gaussian noise around the mean of the distribution µ, according to the current covariance matrix Σ, i.e. x i ∼ N (µ, Σ). The problemdependent fitness of these new individuals (f i) i=1,...,λ is computed, and the top-performing K e individuals, (z i) i=1,...,Ke are used to update the parameters of the distribution as follows: DISPLAYFORM0 DISPLAYFORM1 where (λ i) i=1,...,Ke are weights given to the individuals, commonly chosen with BID10. In the former, each individual is given the same importance, whereas the latter gives more importance to better individuals. BID0. DISPLAYFORM2 A minor difference between CEM and CMA-ES can be found in the update of the covariance matrix. In its standard formulation, CEM uses the new estimate of the mean µ to compute the new Σ, whereas CMA-ES uses the current µ (the one that was used to sample the current generation) as is the case in. We used the latter as BID10 shows it to be more efficient. The algorithm we are using can thus be described either as CEM using the current µ for the estimation of the new Σ, or as CMA-ES without evolutionary paths. The difference being minor, we still call the ing algorithm CEM. Besides, we add some noise in the form of I to the usual covariance update to prevent premature convergence. We choose to have an exponentially decaying, by setting an initial and a final standard deviation, respectively σ init and σ end, initializing to σ init and updating at each iteration with = τ cem + (1 − τ cem)σ end.Note that, in practice Σ can be too large for computing the updates and sampling new individuals. Indeed, if n denotes the number of actor parameters, simply sampling from Σ scales at least in O(n 2.3), which becomes quickly intractable. Instead, we constrain Σ to be diagonal. This means that in our computations, we replace the update in by DISPLAYFORM3 where the square of the vectors denote the vectors of the square of the coordinates. The Deep Deterministic Policy Gradient (DDPG) BID17 and Twin Delayed Deep Deterministic policy gradient (TD3) BID8 algorithms are two off-policy, actorcritic and sample efficient deep RL algorithms. The DDPG algorithm suffers from instabilities partly due to an overestimation bias in critic updates, and is known to be difficult to tune given its sensitivity to hyper-parameter settings. The availability of properly tuned code baselines incorporating several advanced mechanisms improves on the latter issue. The TD3 algorithm rather improves on the former issue, limiting the over-estimation bias by using two critics and taking the lowest estimate of the action values in the update mechanisms. As shown in FIG0, the CEM-RL method combines CEM with either DDPG or TD3, giving rise to two algorithms named CEM-DDPG and CEM-TD3. The mean actor of the CEM population, referred to as π µ, is first initialized with a random actor network. A unique critic network Q π managed by TD3 or DDPG is also initialized. At each iteration, a population of actors is sampled by adding Gaussian noise around the current mean π µ, according to the current covariance matrix Σ. Half of the ing actors are directly evaluated. The corresponding fitness is computed as the cumulative reward obtained during an episode in the environment. Then, for each actor of the other half of the population, the critic is updated using this actor and, reciprocally, the actor follows the direction of the gradient given by the critic Q π for a fixed number of steps. The ing actors are evaluated after this process. The CEM algorithm then takes the top-performing half of the ing global population to compute its new π µ and Σ. The steps performed in the environment used to evaluate all actors in the population are fed into the replay buffer. The critic is trained from that buffer pro rata to the quantity of new information introduced in the buffer at the current generation. For instance, if the population contains 10 individuals, and if each episode lasts 1000 time steps, then 10,000 new samples are introduced in the replay buffer at each generation. The critic is thus trained for 10,000 mini-batches, which are divided into 2000 mini-batches per learning actor. This is a common practice in deep RL algorithms, where one mini-batch update is performed for each step of the actor in the environment. We also choose this number of steps to be the number of gradient steps taken by half of the population at the next iteration. A pseudo-code of CEM-RL is provided in Algorithm 1.In cases where applying the gradient increases the performance of the actor, CEM benefits from this increase by incorporating the corresponding actors in its computations. By contrast, in cases where the gradient steps decrease performance, the ing actors are ignored by CEM, which instead focuses on standard samples around π µ. Those poor samples do not bring new insight on the current distribution of the CEM algorithm, since the gradient steps takes them away from the current distribution. However, since all evaluated actors are filling the replay buffer, the ing experience is still fed to the critic and the future learning actors, providing some supplementary exploration. This approach generates a beneficial flow of information between the deep RL part and the evolutionary part. Indeed, on one hand, good actors found by following the current critic directly improve the evolutionary population. On the other hand, good actors found through evolution fill the replay buffer from which the RL algorithm learns. In that respect, our approach benefits from the same properties as the ERL algorithm BID14 ) depicted in FIG0. But, by contrast with BID14, gradient steps are directly applied to several samples, and using the CEM algorithm makes it possible to use importance mixing, as described in Appendix B. Another difference is that in CEM-RL gradient steps are applied at each iteration whereas in ERL, a deep RL actor is only injected to the population from time to time. One can also see from FIG0 that, in contrast to ERL, CEM-RL does not use any deep RL actor. Other distinguishing properties between ERL and CEM-RL are discussed in the light of empirical in Section 5.2.Finally, given that CMA-ES is generally considered as more sophisticated than CEM, one may wonder why we did not use CMA-ES instead of CEM into the CEM-RL algorithm. Actually, the key contribution of CMA-ES with respect to CEM consists of the evolutionary path mechanism (see Section 3.2), but this mechanism in some inertia in Σ updates, which resists to the beneficial effect of applying RL gradient steps. In this section, we study the CEM-RL algorithm to answer the following questions:• How does the performance of CEM-RL compare to that of CEM and TD3 taken separately?What if we remove the CEM mechanism, ing in a multi-actor TD3? Require: max steps, the maximum number of steps in the environment τ CEM, σ init, σ end and pop size, hyper-parameters of the CEM algorithm γ, τ, lr actor and lr critic, hyper-parameters of DDPG 1: Initialize a random actor π µ, to be the mean of the CEM algorithm 2: Let Σ = σ init I be the covariance matrix of the CEM algorithm 3: Initialize the critic Q π and the target critic Q π t 4: Initialize an empty cyclic replay buffer R 5: total steps, actor steps = 0, 0 6: while total steps < max steps:7:Draw the current population pop from N (π µ, Σ) with importance mixing (see Algorithm 2 in Appendix B)8:for i ← 1 to pop size/2: DISPLAYFORM0 Set the current policy π to pop [i] 10:Initialize a target actor π t with the weights of π 11:Train Q π for 2 * actor steps / pop size mini-batches 12:Train π for actor steps mini-batches 13:Reintroduce the weights of π in pop 14:actor steps = 0 for i ← 1 to pop size:16:Set the current policy π to pop [i] 17:(fitness f, steps s) ← evaluate(π)18:Fill R with the collected experiences 19:actor steps = actor steps + s total steps = total steps + actor steps 20:Update π µ and Σ with the top half of the population (see and in Section 3.2) 21: end while• How does CEM-RL perform compared to ERL? What are the main factors explaining the difference between both algorithms?Additionally, in Appendices B to E, we investigate other aspects of the performance of CEM-RL such as the impact of importance mixing, the addition of action noise or the use of the tanh non-linearity. In order to investigate the above questions, we evaluate the corresponding algorithms in several continuous control tasks simulated with the MUJOCO physics engine and commonly used as policy search benchmarks: HALF-CHEETAH-V2, HOPPER-V2, WALKER2D-V2, SWIMMER-V2 and ANT-V2 BID2.We implemented CEM-RL with the PYTORCH library 1. We built our code around the DDPG and TD3 implementations given by the authors of the TD3 algorithm 2. For the ERL implementation, we used one given by the authors 3.Unless specified otherwise, each curve represents the average over 10 runs of the corresponding quantity, and the variance corresponds to the 68% confidence interval for the estimation of the mean. In all learning performance figures, dotted curves represent medians and the x-axis represents the total number of steps actually performed in the environment, to highlight potential sample efficiency effects, particularly when using importance mixing (see Appendix B).Architectures of the networks are described in Appendix A. Most TD3 and DDPG hyper-parameters were reused from BID8. The only notable difference is the use of tanh non linearities instead of RELU in the actor network, after we spotted that the latter performs better than the former on several environments. We trained the networks with the Adam optimizer BID15, with a learning rate of 1e −3 for both the actor and the critic. The discount rate γ was set to 0.99, and the target weight τ to 5e −3. All populations contained 10 actors, and the standard deviations σ init, σ end and the constant τ cem of the CEM algorithm were respectively set to 1e −3, 1e −5 and 0.95. Finally, the size of the replay buffer was set to 1e 6, and the batch size to 100. We first compare CEM-TD3 to TD3, TD3 and a multi-actor variant of TD3, then CEM-RL to ERL based on several benchmarks. A third section is devoted to additional which have been rejected in appendices to comply with space constraints. In this section, we compare CEM-TD3 to three baselines: our variant of CEM, TD3 and a multi-actor variant of TD3. For TD3 and its multi-actor variant, we report the average of the score of the agent over 10 episodes for every 5000 steps performed in the environment. For CEM and CEM-TD3, we report after each generation the average of the score of the new average individual of the population over 10 episodes. From FIG1, one can see that CEM-TD3 outperforms CEM and TD3 on HALF-CHEETAH-V2, HOPPER-V2 and WALKER2D-V2. On most benchmarks, CEM-TD3 also displays slightly less variance than TD3. Further in Appendix G show that on ANT-V2, CEM-TD3 outperforms CEM and is on par with TD3. More surprisingly, CEM outperforms all other algorithms on SWIMMER-V2, as covered in Appendix E.One may wonder whether the good performance of CEM-TD3 mostly comes from its "ensemble method" nature BID19. Indeed, having a population of actors improves exploration and stabilizes performances by filtering out instabilities that can appear during learning. To answer this question, we performed an ablative study where we removed the CEM mechanism. We considered a population of 5 actors initialized as in CEM-TD3, but then just following the gradient given by the TD3 critic. This algorithm can be seen as a multi-actor TD3 where all actors share the same critic. We reused the hyper-parameters described in Section 5.2. From FIG1, one can see that CEM-TD3 outperforms more or less significantly multi-actor TD3 on all benchmarks, which clearly suggests that the evolutionary part contributes to the performance of CEM-TD3.As a summary, In this section, we compare CEM-RL to ERL. The ERL method using DDPG rather than TD3, we compare it to both CEM-DDPG and CEM-TD3. This makes it possible to isolate the effect of the combination scheme from the improvement brought by TD3 itself. Results are shown in FIG2. We let ERL learn for the same number of steps as in Khadka & Tumer, namely 2 millions on HALF-CHEETAH-V2 and SWIMMER-V2, 4 millions on HOPPER-V2, 6 millions on ANT-V2 and 10 millions on WALKER2D-V2. However, due to limited computational resources, we stop learning with both CEM-RL methods after 1 million steps, hence the constant performance after 1 million steps. Our slightly differ from those of the ERL paper BID14. We explain this difference by two factors. First, the authors only average their over 5 different seeds, whereas we used 10 seeds. Second, the released implementation of ERL may be slightly different from the one used to produce the published 4, raising again the reproducibility issue recently discussed in the reinforcement learning literature BID11. FIG2 shows that after performing 1 million steps, both CEM-RL methods outperform ERL on HALF-CHEETAH-V2, HOPPER-V2 and WALKER2D-V2. We can also see that CEM-TD3 outperforms CEM-DDPG on WALKER2D-V2. On ANT-V2, CEM-DDPG and ERL being on par after 1 million steps, we increased the number of learning steps in CEM-DDPG to 6 millions. The corresponding are shown in FIG0 in Appendix G. Results on SWIMMER-V2 are covered in Appendix E. Table 2: Final performance of ERL, CEM-DDPG and CEM-TD3 on 5 environments. We report the mean ands medians over 10 runs of 1 million steps. For each benchmark, we highlight the of the method with the best mean. One can see that, beyond outperforming ERL, CEM-TD3 outperforms CEM-DDPG on most benchmarks, in terms of final performance, convergence speed, and learning stability. This is especially true for hard environments such as WALKER2D-V2 and ANT-V2. The only exception in SWIMMER-V2, as studied in Appendix E. Table 2 gives the final best of methods used in this Section. The overall is that CEM-RL generally outperforms ERL. In this section, we outline the main messages arising from further studies that have been rejected in appendices in order to comply with space constraints. In Appendix B, we investigate the influence of the importance mixing mechanism over the evolution of performance, for CEM and CEM-RL. Results show that importance mixing has a limited impact on the sample efficiency of CEM-TD3 on the benchmarks studied here, in contradiction with from BID22 obtained using various standard evolutionary strategies. The fact that the covariance matrix Σ moves faster with CEM-RL may explain this , as it prevents the reuse of samples. In Appendix C, we analyze the effect of adding Gaussian noise to the actions of CEM-TD3. Unlike what BID14 suggested using ERL, we did not find any conclusive evidence that action space noise improves performance with CEM-TD3. This may be due to the fact that, as further studied in Appendix D, the evolutionary algorithm in ERL tends to converge to a unique individual, hence additional noise is welcome, whereas evolutionary strategies like CEM more easily maintain some exploration. Indeed, we further investigate the different dynamics of parameter space exploration provided by the ERL and CEM-TD3 algorithms in Appendix D. FIG6 and 7 show that the evolutionary population in ERL tends to collapse towards a single individual, which does not happen with the CEM population due to the sampling method. In Appendix E, we highlight the fact that, on the SWIMMER-V2 benchmark, the performance of the algorithms studied in this paper varies a lot from the performance obtained on other benchmarks. The most likely explanation is that, in SWIMMER-V2, any deep RL method provides a deceptive gradient information which is detrimental to convergence towards efficient actor parameters. In this particular context, ERL better resists to detrimental gradients than CEM-RL, which suggests to design a version of ERL using CEM to improve the population instead of its ad hoc evolutionary algorithm. Finally, in Appendix F, we show that using a tanh non-linearity in the architecture of actors often in significantly stronger performance than using RELU. This strongly suggests performing "neural architecture search" (; BID7 in the context of RL. We advocated in this paper for combining evolutionary and deep RL methods rather than opposing them. In particular, we have proposed such a combination, the CEM-RL method, and showed that in most cases it was outperforming not only some evolution strategies and some sample efficient offpolicy deep RL algorithms, but also another combination, the ERL algorithm. Importantly, despite being mainly an evolutionary method, CEM-RL is competitive to the state-of-the-art even when considering sample efficiency, which is not the case of other deep neuroevolution methods BID24 .Beyond these positive performance , our study raises more fundamental questions. First, why does the simple CEM algorithm perform so well on the SWIMMER-V2 benchmark? Then, our empirical study of importance mixing did not confirm a clear benefit of using it, neither did the effect of adding noise on actions. We suggest explanations for these phenomena, but nailing down the fundamental reasons behind them will require further investigations. Such deeper studies will also help understand which properties are critical in the performance and sample efficiency of policy search algorithms, and define even more efficient policy search algorithms in the future. As suggested in Section 5.2.3, another avenue for future work will consist in designing an ERL algorithm based on CEM rather than on an ad hoc evolutionary algorithm. Finally, given the impact of the neural architecture on our , we believe that a more systemic search of architectures through techniques such as neural architecture search (; BID7 may provide important progress in performance of deep policy search algorithms. This work was supported by the European Commission, within the DREAM project, and has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement N o 640891. We would like to thank Thomas Pierrot for fruitful discussions. A ARCHITECTURE OF THE NETWORKS Our network architectures are very similar to the ones described in BID8 . In particular, the size of the layers remains the same. The only differences resides in the non-linearities. Importance mixing is a specific mechanism designed to improve the sample efficiency of evolution strategies. It was initially introduced in Sun et al. FORMULA1 and consisted in reusing some samples from the previous generation into the current one, to avoid the cost of re-evaluating the corresponding policies in the environment. The mechanism was recently extended in BID22 to reusing samples from any generation stored into an archive. Empirical showed that importance sampling can improve sample efficiency by a factor of ten, and that most of these savings just come from using the samples from the previous generation, as performed by the initial mechanism. A pseudo-code of the importance mixing mechanism is given in Algorithm 2.In CEM, importance mixing is implemented as described in BID22 . By contrast, some adaptation is required in CEM-RL. Actors which take gradient steps can no longer be regarded as sampled from the current distribution of the CEM algorithm. We thus choose to apply importance mixing only to the half of the population which does not receive gradient steps from the RL critic. In practice, only actors which do not take gradient steps are inserted into the actor archive and can be replaced with samples from previous generations. From Figure 4, one can see that in the CEM case, importance mixing introduces some minor instability, without noticeably increasing sample efficiency. On HALF-CHEETAH-V2, SWIMMER-V2 and WALKER2D-V2, performance even decreases when using importance mixing. For CEM-RL, the effect varies greatly from environment to environment, but the gain in sample reuse is almost null if min(1, DISPLAYFORM0 Append z i to the current generation g new 7:Draw z i according to the current pdf p(., θ new)8: DISPLAYFORM1 Append z i to the current generation g new 10:size = |g new | if size ≥ N: go to 1212: if size > N: remove a randomly chosen sample 13: if size < N: fill the generation sampling from p(., DISPLAYFORM0 Figure 4: Learning curves of CEM-TD3 and CEM with and without importance mixing on the HALF-CHEETAH-V2, HOPPER-V2, WALKER2D-V2, SWIMMER-V2 and ANT-V2 benchmarks.as well, though an increase in performance can be seen on SWIMMER-V2. The latter fact is consistent with the finding that the gradient steps are not useful in this environment (see Appendix E). On HOPPER-V2 and HALF-CHEETAH-V2, with and without importance mixing seem to be equivalent. On WALKER2D-V2, importance mixing decreases final performance. On ANT-V2, importance mixing seems to accelerate learning in the beginning, but final performances are equivalent to those of CEM-RL. Thus importance mixing seems to have a limited impact in CEM-TD3.This seems to contradict the obtained in BID22. This may be due to different things. First, the dimensions of the search spaces in the experiments here are much larger than those studied in BID22, which might deteriorate the estimation of the covariance matrices when samples are too correlated. the ones used in BID22. In particular, we can see from FIG1 that CEM is far from solving the environments over one million steps. Perhaps a study over a longer time period would make importance mixing relevant again. Besides, by reusing old samples, the importance mixing mechanism somehow hinders exploration (since we evaluate less new individuals), which might be detrimental in the case of MUJOCO environments. Finally, and most importantly, the use of RL gradient steps accelerates the displacement of the covariance matrix, ing in fewer opportunities for sample reuse. In BID14, the authors indicate that one reason for the efficiency of their approach is that the replay buffer of DDPG gets filled with two types of noisy experiences. On one hand, the buffer gets filled with noisy interactions of the DDPG actor with the environment. This is usually referred to as action space noise. On the other hand, actors with different parameters also fill the buffer, which is more similar to parameter space noise. In CEM-RL, we only use parameter space noise, but it would also be possible to add action space noise. To explore this direction, each actor taking gradient steps performs a noisy episode in the environment. We report final after 1 million steps in TAB9. Learning curves are available in FIG5. Unlike what BID14 suggested, we did not find any conclusive evidence that action space noise improves performance. In CEM-TD3, the CEM part seems to explore enough of the action space on its own. It seems that sampling performed in CEM in sufficient exploration and performs better than adding simple Gaussian noise to the actions. This highlights a difference between using an evolutionary strategy like CEM and an evolutionary algorithm as done in ERL. Evolutionary algorithms tend to converge to a unique individual whereas evolutionary strategies more easily maintain some exploration. These aspects are further studied in Appendix D. In this section, we highlight the difference in policy parameter update dynamics in CEM-RL and ERL. FIG6 displays the evolution of the first two parameters of actor networks during training with CEM-RL and ERL on HALF-CHEETAH-V2. For ERL, we plot the chosen parameters of the DDPG actor with a continuous line, and represent those of the evolutionary actors with dots. For CEM-RL, we represent the chosen parameters of sampled actors with dots, and the gradient steps based on the TD3 critic with continuous lines. The same number of dots is represented for both algorithms. One can see that, in ERL the evolutionary population tends to be much less diverse that in CEM-RL. There are many redundancies in the parameters (dots with the same coordinates), and the population seems to converge to a single individual. On the other hand, there is no such behavior in CEM-RL where each generation introduces completely new samples. As a consequence, parameter space exploration looks better in the CEM-RL algorithm. To further study this loss of intra-population diversity in ERL, we perform 10 ERL runs and report in FIG7 an histogram displaying the distribution of the population-wise similarity with respect to the populations encountered during learning. We measure this similarity as the average percentage of parameters shared between two different individuals of the said population. The indicate that around 55% of populations encountered during a run of ERL display a population-similarity of above 80%. Results are averaged over 10 runs. As usual, the variance corresponds to the 68% confidence interval for the estimation of the mean. One can also see the difference in how both methods use the gradient information of their respective deep RL part. In the case of ERL, the parameters of the population concentrate around those of the DDPG actor. Each 10 generations, its parameters are introduced into the population, and since DDPG is already efficient alone on HALF-CHEETAH-V2, those parameters quickly spread into the population. Indeed, according to BID14, the ing DDPG actor is the elite of the population 80% of the time, and is introduced into the population 98% of the time. This integration is however passive: the direction of exploration does not vary much after introducing the DDPG agent. CEM-RL integrates this gradient information differently. The short lines emerging from dots, which represent gradient steps performed by half of the actors, act as scouts. Once CEM becomes aware of better solutions that can be found in a given direction, the sampling of the next population is modified so as to favor this promising direction. CEM is thus pro-actively exploring in the good directions it has been fed with. Experiments on the SWIMMER-V2 benchmark give that differ a lot from the on other benchmarks, hence they are covered separately here. Figure8a shows that CEM outperforms TD3, CEM-TD3, multi-actor TD3. Besides, as shown in FIG8, ERL outperforms CEM-DDPG, which itself outperforms CEM-TD3. All these findings seem to show that being better at RL makes you worse at SWIMMER-V2. The most likely explanation is that, in SWIMMER-V2, any deep RL method provides a deceptive gradient information which is detrimental to convergence towards efficient actor parameters. This could already be established from the of BID14, where the evolution algorithm alone produced on par with the ERL algorithm, showing that RL-based actors were just ignored. In this particular context, the actors using TD3 gradient being deteriorated by the deceptive gradient effect, CEM-RL is behaving as a CEM with only half a population, thus it is less efficient than the standard CEM algorithm. By contrast, ERL better resists than CEM-RL to the same issue. Indeed, if the actor generated by DDPG does not perform better than the evolutionary population, then this actor is just ignored, and the evolutionary part behaves as usual, without any loss in performance. In practice, Khadka & Tumer note that on SWIMMER-V2, the DDPG actor was rejected 76% of the time. Finally, by comparing CEM and ERL from FIG8 and FIG8, one can conclude that on this benchmark, the evolutionary part of ERL behaves on par with CEM alone. This is at odds with premature convergence effects seen in the evolutionary part of ERL, as studied in more details in Appendix D. From all these insights, the SWIMMER-V2 environment appears particularly interesting, as we are not aware of any deep RL method capable of solving it quickly and reliably. In this section, we explore the impact on performance of the type of non-linearities used in the actor of CEM-TD3. TAB9 reports the of CEM-TD3 using RELU non-linearities between the linear layers, instead of tanh. Figure 9 displays the learning performance of CEM-TD3 and CEM on benchmarks, using either the RELU or the tanh nonlinearity in the actors. Results indicate that on some benchmarks, changing from tanh to RELU can cause a huge drop in performance. This is particularly obvious in the ANT-V2 benchmark, where the average performance drops by 46%. FIG9 (f) shows that, for the CEM algorithm on the SWIMMER-V2 benchmark, using RELU also causes a 60% performance drop. As previously reported in the literature BID11, this study suggests that network architectures can have a large impact on performance. G ADDITIONAL ON ANT-V2 FIG0 represents the learning performance of CEM, TD3, multi-actor TD3, CEM-DDPG and CEM-TD3 on the ANT-V2 benchmark. It is discussed in the main text.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a new combination of evolution strategy and deep reinforcement learning which takes the best of both worlds


Question: Represent the article for finding a single-sentence TLDR!

Question: Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games. However, developing practical methods to balance exploration and exploitation in complex domains remains largely unsolved. Thompson Sampling and its extension to reinforcement learning provide an elegant approach to exploration that only requires access to posterior samples of the model. At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical. Thus, it is attractive to consider approximate Bayesian neural networks in a Thompson Sampling framework. To understand the impact of using an approximate posterior on Thompson Sampling, we benchmark well-established and recently developed methods for approximate posterior sampling combined with Thompson Sampling over a series of contextual bandit problems. We found that many approaches that have been successful in the supervised learning setting underperformed in the sequential decision-making scenario. In particular, we highlight the challenge of adapting slowly converging uncertainty estimates to the online setting. Recent advances in reinforcement learning have sparked renewed interest in sequential decision making with deep neural networks. Neural networks have proven to be powerful and flexible function approximators, allowing one to learn mappings directly from complex states (e.g., pixels) to estimates of expected return. While such models can be accurate on data they have been trained on, quantifying model uncertainty on new data remains challenging. However, having an understanding of what is not yet known or well understood is critical to some central tasks of machine intelligence, such as effective exploration for decision making. A fundamental aspect of sequential decision making is the exploration-exploitation dilemma: in order to maximize cumulative reward, agents need to trade-off what is expected to be best at the moment, (i.e., exploitation), with potentially sub-optimal exploratory actions. Solving this trade-off in an efficient manner to maximize cumulative reward is a significant challenge as it requires uncertainty estimates. Furthermore, exploratory actions should be coordinated throughout the entire decision making process, known as deep exploration, rather than performed independently at each state. Thompson Sampling and its extension to reinforcement learning, known as Posterior Sampling, provide an elegant approach that tackles the exploration-exploitation dilemma by maintaining a posterior over models and choosing actions in proportion to the probability that they are optimal. Unfortunately, maintaining such a posterior is intractable for all but the simplest models. As such, significant effort has been dedicated to approximate Bayesian methods for deep neural networks. These range from variational methods BID17 BID6 BID23 to stochastic minibatch Markov Chain Monte Carlo (; ; BID25 BID1 BID26, among others. Because the exact posterior is intractable, evaluating these approaches is hard. Furthermore, these methods are rarely compared on benchmarks that measure the quality of their estimates of uncertainty for downstream tasks. To address this challenge, we develop a benchmark for exploration methods using deep neural networks. We compare a variety of well-established and recent Bayesian approximations under the lens of Thompson Sampling for contextual bandits, a classical task in sequential decision making. All code and implementations to reproduce the experiments will be available open-source, to provide a reproducible benchmark for future development. 1 Exploration in the context of reinforcement learning is a highly active area of research. Simple strategies such as epsilon-greedy remain extremely competitive . However, a number of promising techniques have recently emerged that encourage exploration though carefully adding random noise to the parameters (; BID12 BID13 or bootstrap sampling before making decisions. These methods rely explicitly or implicitly on posterior sampling for exploration. In this paper, we investigate how different posterior approximations affect the performance of Thompson Sampling from an empirical standpoint. For simplicity, we restrict ourselves to one of the most basic sequential decision making scenarios: that of contextual bandits. No single algorithm bested the others in every bandit problem, however, we observed some general trends. We found that dropout, injecting random noise, and bootstrapping did provide a strong boost in performance on some tasks, but was not able to solve challenging synthetic exploration tasks. Other algorithms, like Variational Inference, Black Box α-divergence, and minibatch Markov Chain Monte Carlo approaches, strongly couple their complex representation and uncertainty estimates. This proves problematic when decisions are made based on partial optimization of both, as online scenarios usually require. On the other hand, making decisions according to a Bayesian linear regression on the representation provided by the last layer of a deep network offers a robust and easy-to-tune approach. It would be interesting to try this approach on more complex reinforcement learning domains. In Section 2 we discuss Thompson Sampling, and present the contextual bandit problem. The different algorithmic approaches that approximate the posterior distribution fed to Thompson Sampling are introduced in Section 3, while the linear case is described in Section 4. The main experimental are presented in Section 5, and discussed in Section 6. Finally, Section 7 concludes. The contextual bandit problem works as follows. At time t = 1,..., n a new context X t ∈ R d arrives and is presented to algorithm A. The algorithm -based on its internal model and X t -selects one of the k available actions, a t. Some reward r t = r t (X t, a t) is then generated and returned to the algorithm, that may update its internal model with the new data. At the end of the process, the reward for the algorithm is given by r = n t=1 r t, and cumulative regret is defined as R A = E[r * − r], where r * is the cumulative reward of the optimal policy (i.e., the policy that always selects the action with highest expected reward given the context). The goal is to minimize R A.The main research question we address in this paper is how approximated model posteriors affect the performance of decision making via Thompson Sampling (Algorithm 1) in contextual bandits. We study a variety of algorithmic approaches to approximate a posterior distribution, together with different empirical and synthetic data problems that highlight several aspects of decision making. We consider distributions π over the space of parameters that completely define a problem instance θ ∈ Θ. For example, θ could encode the reward distributions of a set of arms in the multi-armed bandit scenario, or -more generally-all the parameters of an MDP in reinforcement learning. Thompson Sampling is a classic algorithm which requires only that one can sample from the posterior distribution over plausible problem instances (for example, values or rewards). At each round, it draws a sample and takes a greedy action under the optimal policy for the sample. The posterior distribution is then updated after the of the action is observed. Thompson Sampling has been shown to be extremely effective for bandit problems both in practice BID9 BID16 and theory BID0. It is especially appealing for deep neural networks as one rarely has access to the full posterior but can often approximately sample from it. In this section, we describe the different algorithmic design principles that we considered in our simulations of Section 5. These algorithms include linear methods, Neural Linear and Neural Greedy, variational inference, expectation-propagation, dropout, Monte Carlo methods, bootstrapping, direct noise injection, and Gaussian Processes. In FIG10 in the appendix, we visualize the posteriors of the nonlinear algorithms on a synthetic one dimensional problem. Linear Methods We apply well-known closed-form updates for Bayesian linear regression for exact posterior inference in linear models BID5. We provide the specific formulas below, and note that they admit a computationally-efficient online version. We consider exact linear posteriors as a baseline; i.e., these formulas compute the posterior when the data was generated according to Y = X T β + where ∼ N (0, σ 2), and Y represents the reward. Importantly, we model the joint distribution of β and σ 2 for each action. Sequentially estimating the noise level σ 2 for each action allows the algorithm to adaptively improve its understanding of the volume of the hyperellipsoid of plausible β's; in general, this leads to a more aggressive initial exploration phase (in both β and σ 2).The posterior at time t for action i, after observing X, Y, is DISPLAYFORM0, where we assume σ 2 ∼ IG(a t, b t), and β | σ 2 ∼ N (µ t, σ 2 Σ t), an Inverse Gamma and Gaussian distribution, respectively. Their parameters are given by DISPLAYFORM1 DISPLAYFORM2 We set the prior hyperparameters to µ 0 = 0, and Λ 0 = λ Id, while a 0 = b 0 = η > 1. It follows that initially, for σ We consider two approximations to motivated by function approximators where d is large. While posterior distributions or confidence ellipsoids should capture dependencies across parameters as shown above (say, a dense Σ t), in practice, computing the correlations across all pairs of parameters is too expensive, and diagonal covariance approximations are common. For linear models it may still be feasible to exactly compute, whereas in the case of Bayesian neural networks, unfortunately, this may no longer be possible. Accordingly, we study two linear approximations where Σ t is diagonal. Our goal is to understand the impact of such approximations in the simplest case, to properly set our expectations for the loss in performance of equivalent approximations in more complex approaches, like mean-field variational inference or Stochastic Gradient Langevin Dynamics. Assume for simplicity the noise standard deviation is known. In FIG1, for d = 2, we see the posterior distribution β t ∼ N (µ t, Σ t) of a linear model based on, in green, together with two diagonal approximations. Each approximation tries to minimize a different objective. In blue, the PrecisionDiag posterior approximation finds the diagonalΣ ∈ R d×d minimizing KL(N (µ t,Σ) || N (µ t, Σ t)), like in mean-field variational inference. In particular,Σ = Diag(Σ −1 t) −1. On the other hand, in orange, the Diag posterior approximation finds the diagonal matrixΣ minimizing KL(N (µ t, Σ t) || N (µ t,Σ)) instead. In this case, the solution is simplyΣ = Diag(Σ t).We add linear baselines that do not model the uncertainty in the action noise σ 2. In addition, we also consider simple greedy and epsilon greedy linear baselines (i.e., not based on Thompson Sampling). The main problem linear algorithms face is their lack of representational power, which they complement with accurate uncertainty estimates. A natural attempt at getting the best of both worlds consists in performing a Bayesian linear regression on top of the representation of the last layer of a neural network, similarly to. The predicted value v i for each action a i is given by v i = β T i z x, where z x is the output of the last hidden layer of the network for context x. While linear methods directly try to regress values v on x, we can independently train a deep net to learn a representation z, and then use a Bayesian linear regression to regress v on z, obtain uncertainty estimates on the β's, and make decisions accordingly via Thompson Sampling. Note that we do not explicitly consider the weights of the linear output layer of the network to make decisions; further, the network is only used to find good representations z. In addition, we can update the network and the linear regression at different time-scales. It makes sense to keep an exact linear regression (as in FORMULA1 and FORMULA2) at all times, adding each new data point as soon as it arrives. However, we only update the network after a number of points have been collected. In our experiments, after updating the network, we perform a forward pass on all the training data to obtain z x, which is then fed to the Bayesian regression. In practice this may be too expensive, and z could be updated periodically with online updates on the regression. We call this algorithm Neural Linear. Neural Greedy We refer to the algorithm that simply trains a neural network and acts greedily (i.e., takes the action whose predicted score for the current context is highest) as RMS, as we train it using the RMSProp optimizer. This is our non-linear baseline, and we tested several versions of it (based on whether the training step was decayed, reset to its initial value for each re-training or not, and how long the network was trained for). We also tried the -greedy version of the algorithm, where a random action was selected with probability for some decaying schedule of.Variational Inference Variational approaches approximate the posterior by finding a distribution within a tractable family that minimizes the KL divergence to the posterior BID20. These approaches formulate and solve an optimization problem, as opposed, for example, to sampling methods like MCMC BID22 ). Typically (and in our experiments), the posterior is approximated by a mean-field or factorized distribution where strong independence assumptions are made. For instance, each neural network weight can be modeled via a -conditionally independent-Gaussian distribution whose mean and variance are estimated from data. Recent advances have scaled these approaches to estimate the posterior of neural networks with millions of parameters BID6. A common criticism of variational inference is that it underestimates uncertainty (e.g., BID5), which could lead to under-exploration. The family of expectation-propagation algorithms (; BID30 a) is based on the message passing framework . They iteratively approximate the posterior by updating a single approximation factor (or site) at a time, which usually corresponds to the likelihood of one data point. The algorithm sequentially minimizes a set of local KL divergences, one for each site. Most often, and for computational reasons, likelihoods are chosen to lie in the exponential family. In this case, the minimization corresponds to moment matching. See BID14 for further details. We focus on methods that directly optimize the global EP objective via stochastic gradient descent, as, for instance, Power EP BID28. In particular, in this work, we implement the black-box α-divergence minimization algorithm BID19, where local parameter sharing is applied to the Power EP energy function. Note that different values of α ∈ R\{0} correspond to common algorithms: α = 1 to EP, and α → 0 to Variational Bayes. The optimal α value is problem-dependent BID19.Dropout Dropout is a training technique where the output of each neuron is independently zeroed out with probability p at each forward pass . Once the network has been trained, dropout can still be used to obtain a distribution of predictions for a specific input. Following the best action with respect to the random dropout prediction can be interpreted as an implicit form of Thompson sampling. Dropout can be seen as optimizing a variational objective BID23 BID13 BID21.Monte Carlo Monte Carlo sampling remains one of the simplest and reliable tools in the Bayesian toolbox. Rather than parameterizing the full posterior, Monte Carlo methods estimate the posterior through drawing samples. This is naturally appealing for highly parameterized deep neural networks for which the posterior is intractable in general and even simple approximations such as multivariate Gaussian are too expensive (i.e. require computing and inverting a covariance matrix over all parameters). Among Monte Carlo methods, Hamiltonian Monte Carlo (HMC) is often regarded as a gold standard algorithm for neural networks as it takes advantage of gradient information and momentum to more effectively draw samples. However, it remains unfeasible for larger datasets as it involves a Metropolis accept-reject step that requires computing the log likelihood over the whole data set. A variety of methods have been developed to approximate HMC using mini-batch stochastic gradients. These Stochastic Gradient Langevin Dynamics (SGLD) methods add Gaussian noise to the model gradients during stochastic gradient updates in such a manner that each update in an approximate sample from the posterior. Different strategies have been developed for augmenting the gradients and noise according to a preconditioning matrix. BID25 show that a preconditioner based on the RMSprop algorithm performs well on deep neural networks. suggested using the Fisher information matrix as a preconditioner in SGLD. Unfortunately the approximations of SGLD hold only if the learning rate is asymptotically annealed to zero. BID1 introduced Stochastic Gradient Fisher Scoring to elegantly remove this requirement by preconditioning according to the Fisher information (or a diagonal approximation thereof). BID26 develop methods for approximately sampling from the posterior using a constant learning rate in stochastic gradient descent and develop a prescription for a stable version of SGFS. We evaluate the diagonal-SGFS and constant-SGD algorithms from BID26 in this work. Specifically for constant-SGD we use a constant learning rate for stochastic gradient descent, where the learning rate is given by = 2 S N BB T where S is the batch size, N the number of data points and BB T is an online average of the diagonal empirical Fisher information matrix. For Stochastic Gradient Fisher Scoring we use the following stochastic gradient update for the model parameters θ at step t: DISPLAYFORM0 where we take the noise covariance EE T to also be BB DISPLAYFORM1 Bootstrap A simple empirical approach to approximate the sampling distribution of any estimator is the Bootstrap BID10. The main idea is to simultaneously train q models, where each model i is based on a different dataset D i. When all the data D is available in advance, D i is typically created by sampling |D| elements from D at random with replacement. In our case, however, the data grows one example at a time. Accordingly, we set a parameter p ∈, and append the new datapoint to each D i independently at random with probability p. In order to emulate Thompson Sampling, we sample a model uniformly at random (i.e., with probability p i = 1/q.) and take the action predicted to be best by the sampled model. We mainly tested cases q = 5, 10 and p = 0.8, 1.0, with neural network models. Note that even when p = 1 and the datasets are identical, the random initialization of each network, together with the randomness from SGD, lead to different predictions. Direct Noise Injection Parameter-Noise is a recently proposed approach for exploration in deep RL that has shown promising . The training updates for the network are unchanged, but when selecting actions, the network weights θ are perturbed with isotropic Gaussian noise. Crucially, the network uses layer normalization BID3, which ensures that all weights are on the same scale. The magnitude of the Gaussian noise is adjusted so that the overall effect of the perturbations is similar in scale to -greedy with a linearly decaying schedule (see for details). Because the perturbations are done on the model parameters, we might hope that the actions produced by the perturbations are more sensible than -greedy. Bayesian Non-parametric Gaussian processes are a gold-standard method for modeling distributions over non-linear continuous functions. It can be shown that, in the limit of infinite hidden units and under a Gaussian prior, a Bayesian neural network converges to a Gaussian process . As such, GPs would appear to be a natural baseline. Unfortunately, standard GPs computationally scale cubically in the number of observations, limiting their applicability to relatively small datasets. There are a wide variety of methods to approximate Gaussian processes using, for example, pseudo-observations or variational inference . We implemented both standard and sparse GPs but only report the former due to similar performance. For the standard GP, due to the scaling issue, we stop adding inputs to the GP after 1000 observations. This performed significantly better than randomly sampling inputs. Our implementation is a multi-task Gaussian process BID7 with a linear and Matern 3/2 product kernel over the inputs and an exponentiated quadratic kernel over latent vectors for the different tasks. The hyperparameters of this model and the latent task vectors are optimized over the GP marginal likelihood. This allows the model to learn correlations between the outputs of the model. Specifically, the covariance function K(·) of the GP is given by: DISPLAYFORM2 and the task kernel between tasks t and l are DISPLAYFORM3 2 ) where v l indexes the latent vector for task l and r λ (x,x) = |(x λ) − (x λ)|. The length-scales, λ m and λ l, and amplitude parameters α, β are optimized via the log marginal likelihood. For the sparse version we used a Sparse Variational GP BID18 with the same kernel and with 300 inducing points, trained via minibatch stochastic gradient descent. In this section, we illustrate some of the subtleties that arise when uncertainty estimates drive sequential decision-making using simple linear examples. There is a fundamental difference between static and dynamic scenarios. In a static scenario, e.g. supervised learning, we are given a model family Θ (like the set of linear models, trees, or neural networks with specific dimensions), a prior distribution π 0 over Θ, and some observed data D that -importantly-is assumed i.i.d. Our goal is to return an approximate posterior distribution: DISPLAYFORM0 We define the quality of our approximation by means of some distance d(π, π).On the other hand, in dynamic settings, our estimate at time t, sayπ t, will be used via some mechanism M, in this case Thompson sampling, to collect the next data-point, which is then appended to D t. In this case, the data-points in D t are no longer independent. D t will now determine two distributions: the posterior given the data that was actually observed, π t+1 = P(θ | D t), and our new estimateπ t+1. When the goal is to make good sequential decisions in terms of cumulative regret, the distance d(π t, π t) is in general no longer a definitive proxy for performance. For instance, a poorly-approximated decision boundary could lead an algorithm, based onπ, to get stuck repeatedly selecting a single sub-optimal action a. After collecting lots of data for that action,π t and π t could start to agree (to their capacity) on the models that explain what was observed for a, while both would stick to something close to the prior regarding the other actions. At that point, d(π t, π t) may show relatively little disagreement, but the regret would already be terrible. Let π * t be the posterior distribution P(θ | D t) under Thompson Sampling's assumption, that is, data was always collected according to π * j for j < t. We follow the idea thatπ t being close to π * t for all t leads to strong performance. However, this concept is difficult to formalize: once different decisions are made, data for different actions is collected and it is hard to compare posterior distributions. We illustrate the previous points with a simple example, see FIG2. Data is generated according to a bandit with k = 6 arms. For a given context X ∼ N (µ, Σ), the reward obtained by pulling arm i follows a linear model r i,X = X T β i + with ∼ N (0, σ DISPLAYFORM1 can be exactly computed using the standard Bayesian linear regression formulas presented in Section 3. We set the contextual dimension d = 20, and the prior to be β ∼ N (0, λ I d), for λ > 0.In FIG2, we show the posterior distribution for two dimensions of β i for each arm i after n = 500 pulls. In particular, in FIG2, two independent runs of Thompson Sampling with their posterior distribution are displayed in red and green. While strongly aligned, the estimates for some arms disagree (especially for arms that are best only for a small fraction of the contexts, like Arm 2 and 3, where fewer data-points are available). In FIG2, we also consider Thompson Sampling with an approximate posterior with diagonal covariance matrix, Diag in red, as defined in Section 3. Each algorithm collects its own data based on its current posterior (or approximation). In this case, the posterior disagreement after n = 500 decisions is certainly stronger. However, as shown in FIG2, if we computed the approximate posterior with a diagonal covariance matrix based on the data collected by the actual posterior, the disagreement would be reduced as much as possible within the approximation capacity (i.e., it still cannot capture correlations in this case). FIG2 shows then the effect of the feedback loop. We look next at the impact that this mismatch has on regret. We illustrate with a similar example how inaccurate posteriors sometimes lead to quite different behaviors in terms of regret. In FIG1, we see the posterior distribution β ∼ N (µ, Σ) of a linear model in green, together with the two diagonal linear approximations introduced in Section 3: the Diag (in orange) and the PrecisionDiag (in blue) approximations, respectively. We now assume there are k linear arms, β i ∈ R d for i = 1,..., k, and decisions are made according to the posteriors in FIG1. In FIG1 we plot the regret of Thompson Sampling when there are k = 20 arms, for both d = 15 and d = 30. We see that, while the PrecisionDiag approximation does even outperform the actual posterior, the diagonal covariance approximation truly suffers poor regret when we increase the dimension d, as it is heavily penalized by simultaneously over-exploring in a large number of dimensions and repeateadly acting according to implausible models. In this section, we present the simulations and outcomes of several synthetic and real-world data bandit problems with each of the algorithms introduced in Section 3. In particular, we first explain how the simulations were set up and run, and the metrics we report. We then split the experiments according to how data was generated, and the underlying models fit by the algorithms from Section 3. We run the contextual bandit experiments as described at the beginning of Section 2, and discuss below some implementation details of both experiments and algorithms. A detailed summary of the key parameters used for each algorithm can be found in Table 2 in the appendix. Neural Network Architectures All algorithms based on neural networks as function approximators share the same architecture. In particular, we fit a simple fully-connected feedforward network with two hidden layers with 100 units each and ReLu activations. The input of the network has dimension d (same as the contexts), and there are k outputs, one per action. Note that for each training point (X t, a t, r t) only one action was observed (and algorithms usually only take into account the loss corresponding to the prediction for the observed action).Updating Models A key question is how often and for how long models are updated. Ideally, we would like to train after each new observation and for as long as possible. However, this may limit the applicability of our algorithms in online scenarios where decisions must be made immediately. We update linear algorithms after each time-step by means of and. For neural networks, the default behavior was to train for t s = 20 or 100 mini-batches every t f = 20 timesteps. 2 The size of each mini-batch was 512. We experimented with increasing values of t s, and it proved essential for some algorithms like variational inference approaches. See the details in Table 2.Metrics We report two metrics: cumulative regret and simple regret. We approximate the latter as the mean cumulative regret in the last 500 time-steps, a proxy for the quality of the final policy (see further discussion on pure exploration settings, BID8). Cumulative regret is computed based on the best expected reward, as is standard. For most real datasets (Statlog, Covertype, Jester, Adult, Census, and Song), the rewards were deterministic, in which case, the definition of regret also corresponds to the highest realized reward (i.e., possibly leading to a hard task, which helps to understand why in some cases all regrets look linear). We reshuffle the order of the contexts, and rerun the experiment 50 times to obtain the cumulative regret distribution and report its statistics. Hyper-Parameter Tuning Deep learning methods are known to be very sensitive to the selection of a wide variety of hyperparameters, and many of the algorithms presented are no exception. Moreover, that choice is known to be highly dataset dependent. Unfortunately, in the bandits scenario, we commonly do not have access to each problem a-priori to perform tuning. For the vast majority of algorithms, we report the outcome for three versions of the algorithm defined as follows. First, we use one version where hyper-parameters take values we guessed to be reasonable a-priori. Then, we add two additional instances whose hyper-parameters were optimized on two different datasets via Bayesian Optimization. For example, in the case of Dropout, the former version is named Dropout, while the optimized versions are named Dropout-MR (using the Mushroom dataset) and Dropout-SL (using the Statlog dataset) respectively. Some algorithms truly benefit from hyper-parameter DISPLAYFORM0 Figure 3: Wheel bandits for increasing values of δ ∈. Optimal action for blue, red, green, black, and yellow regions, are actions 1, 2, 3, 4, and 5, respectively.optimization, while others do not show remarkable differences in performance; the latter are more appropriate in settings where access to the real environment for tuning is not possible in advance. Buffer After some experimentation, we decided not to use a data buffer as evidence of catastrophic forgetting was observed, and datasets are relatively small. Accordingly, all observations are sampled with equal probability to be part of a mini-batch. In addition, as is standard in bandit algorithms, each action was initially selected s = 3 times using round-robin independently of the context. We evaluated the algorithms on a range of bandit problems created from real-world data. In particular, we test on the Mushroom, Statlog, Covertype, Financial, Jester, Adult, Census, and Song datasets (see Appendix Section A for details on each dataset and bandit problem). They exhibit a broad range of properties: small and large sizes, one dominating action versus more homogeneous optimality, learnable or little signal, stochastic or deterministic rewards, etc. For space reasons, the outcome of some simulations are presented in the Appendix. The Statlog, Covertype, Adult, and Census datasets were originally tested in BID11. We summarize the final cumulative regret for Mushroom, Statlog, Covertype, Financial, and Jester datasets in TAB0. In Figure 5 at the appendix, we show a box plot of the ranks achieved by each algorithm across the suite of bandit problems (see Appendix Table 6 and 7 for the full ). As most of the algorithms from Section 3 can be implemented for any model architecture, in this subsection we use linear models as a baseline comparison across algorithms (i.e., neural networks that contain a single linear layer). This allows us to directly compare the approximate methods against methods that can compute the exact posterior. The specific hyper-parameter configurations used in the experiments are described in TAB2 in the appendix. Datasets are the same as in the previous subsection. The cumulative and simple regret are provided in appendix Tables 4 and 5. Some of the real-data problems presented above do not require significant exploration. We design an artificial problem where the need for exploration is smoothly parameterized. The wheel bandit is defined as follows (see Figure 3). Set d = 2, and δ ∈, the exploration parameter. Contexts are sampled uniformly at random in the unit circle in R 2, X ∼ U (D). There are k = 5 possible actions. The first action a 1 always offers reward r ∼ N (µ 1, σ 2), independently of the context. On the other hand, for contexts such that X ≤ δ, i.e. inside the blue circle in Figure 3, the other four actions are equally distributed and sub-optimal, with r ∼ N (µ 2, σ 2) for µ 2 < µ 1. When X > δ, we are outside the blue circle, and only one of the actions a 2,..., a 5 is optimal depending on the sign of context components X = (X 1, X 2). If X 1, X 2 > 0, action 2 is optimal. If X 1 > 0, X 2 < 0, action 3 is optimal, and so on. Non-optimal actions still deliver r ∼ N (µ 2, σ 2) in this region, except a 1 whose mean reward is always µ 1, while the optimal action provides r ∼ N (µ 3, σ 2), with µ 3 µ 1. We set µ 1 = 1.2, µ 2 = 1.0, µ 3 = 50.0, and σ = 0.01. Note that the probability of a context randomly falling in the high-reward region is 1 − δ 2 (not blue). The difficulty of the problem increases with δ, and we expect algorithms to get stuck repeatedly selecting action a 1 for large δ. The problem can be easily generalized for d > 2. Results are shown in Table 9. 100.00 ± 0.15 100.00 ± 0.03 100.00 ± 0.01 100.00 ± 1.48 100.00 ± 1.01 Overall, we found that there is significant room for improvement in uncertainty estimation for neural networks in sequential decision-making problems. First, unlike in supervised learning, sequential decision-making requires the model to be frequently updated as data is accumulated. As a , methods that converge slowly are at a disadvantage because we must truncate optimization to make the method practical for the online setting. In these cases, we found that partially optimized uncertainty estimates can lead to catastrophic decisions and poor performance. Second, and while it deserves further investigation, it seems that decoupling representation learning and uncertainty estimation improves performance. The NeuralLinear algorithm is an example of this decoupling. With such a model, the uncertainty estimates can be solved for in closed form (but may be erroneous due to the simplistic model), so there is no issue with partial optimization. We suspect that this may be the reason for the improved performance. In addition, we observed that many algorithms are sensitive to their hyperparameters, so that best configurations are problem-dependent. Finally, we found that in many cases, the inherit randomness in Stochastic Gradient Descent provided sufficient exploration. Accordingly, in some scenarios it may be hard to justify the use of complicated (and less transparent) variations of simple methods. However, Stochastic Gradient Descent is by no The suffix of the BBB legend label indicates the number of training epochs in each training step. We emphasize that in this evaluation, all algorithms use the same family of models (i.e., linear). While PrecisionDiag exactly solves the mean field problem, BBB relies on partial optimization via SGD. As the number of training epochs increases, BBB improves performance, but is always outperformed by PrecisionDiag.means always enough: in our synthetic exploration-oriented problem (the Wheel bandit) additional exploration was necessary. Next, we discuss our main findings for each class of algorithms. Linear Methods. Linear methods offer a reasonable baseline, surprisingly strong in many cases. While their representation power is certainly a limiting factor, their ability to compute informative uncertainty measures seems to payoff and balance their initial disadvantage. They do well in several datasets, and are able to react fast to unexpected or extreme rewards (maybe as single points can have a heavy impact in fitted models, and their updates are immediate, deterministic, and exact). Some datasets clearly need more complex non-linear representations, and linear methods are unable to efficiently solve those. In addition, linear methods obviously offer computational advantages, and it would be interesting to investigate how their performance degrades when a finite data buffer feeds the estimates as various real-world online applications may require (instead of all collected data).In terms of the diagonal linear approximations described in Section 3, we found that diagonalizing the precision matrix (as in mean-field Variational Inference) performs dramatically better than diagonalizing the covariance matrix. NeuralLinear. The NeuralLinear algorithm sits near a sweet spot that is worth further studying. In general it seems to improve the RMS neural network it is based on, suggesting its exploration mechanisms add concrete value. We believe its main strength is that it is able to simultaneously learn a data representation that greatly simplifies the task at hand, and to accurately quantify the uncertainty over linear models that explain the observed rewards in terms of the proposed representation. While the former process may be noisier and heavily dependent on the amount of training steps that were taken and available data, the latter always offers the exact solution to its approximate parent problem. This, together with the partial success of linear methods with poor representations, may explain its promising . In some sense, it knows what it knows. In the Wheel problem, which requires increasingly good exploration mechanisms, NeuralLinear is probably the best algorithm. Its performance is almost an order of magnitude better than any RMS algorithm (and its spinoffs, like Bootstrapped NN, Dropout, or Parameter Noise), and all greedy linear approaches. On the other hand, it is able to successfully solve problems that require non-linear representations (as Statlog or Covertype) where linear approaches fail. In addition, the algorithm is remarkably easy to tune, and robust in terms of hyper-parameter configurations. While conceptually simple, its deployment to large scale systems may involve some technical difficulties; mainly, to update the Bayesian estimates when the network is re-trained. We believe, however, standard solutions to similar problems (like running averages) could greatly mitigate these issues. In our experiments and compared to other algorithms, as shown in Table 8, NeuralLinear is fast from a computational standpoint. Variational Inference. Overall, Bayes By Backprop performed poorly, ranking in the bottom half of algorithms across datasets TAB0. To investigate if this was due to underestimating uncertainty (as variational methods are known to BID5), to the mean field approximation, or to stochastic optimization, we applied BBB to a linear model, where the mean field optimization problem can be solved in closed form FIG5. We found that the performance of BBB slowly improved as the number of training epochs increased, but underperformed compared to the exact mean field solution. Moreover, the difference in performance due to the number of training steps dwarfed the difference between the mean field solution and the exact posterior. This suggests that it is not sufficient to partially optimize the variational parameters when the uncertainty estimates directly affect the data being collected. In supervised learning, optimizing to convergence is acceptable, however in the online setting, optimizing to convergence at every step incurs unreasonable computational cost. Expectation-Propagation. The performance of Black Box α-divergence algorithms was poor. Because this class of algorithms is similar to BBB (in fact, as α → 0, it converges to the BBB objective), we suspect that partial convergence was also the cause of their poor performance. We found these algorithms to be sensitive to the number of training steps between actions, requiring a large number to achieve marginal performance. Their terrible performance in the Mushroom bandit is remarkable, while in the other datasets they perform slightly worse than their variational inference counterpart. Given the successes of Black Box α-divergence in other domains BID19, investigating approaches to sidestep the slow convergence of the uncertainty estimates is a promising direction for future work. Monte Carlo. Constant-SGD comes out as the winner on Covertype, which requires non-linearity and exploration as evidenced by performance of the linear baseline approaches TAB0 ). The method is especially appealing as it does not require tuning learning rates or exploration parameters. SGFS, however, performs better on average. The additional injected noise in SGFS may cause the model to explore more and thus perform better, as shown in the Wheel Bandit problem where SGFS strongly outperforms Constant-SGD.Bootstrap. The bootstrap offers significant gains with respect to its parent algorithm (RMS) in several datasets. Note that in Statlog one of the actions is optimal around 80% of the time, and the bootstrapped predictions may help to avoid getting stuck, something from which RMS methods may suffer. In other scenarios, the randomness from SGD may be enough for exploration, and the bootstrap may not offer important benefits. In those cases, it might not justify the heavy computational overhead of the method. We found it surprising that the optimized versions of BootstrappedNN decided to use only q = 2 and q = 3 networks respectively (while we set its value to q = 10 in the manually tuned version, and the extra networks did not improve performance significantly). Unfortunately, Bootstrapped NNs were not able to solve the Wheel problem, and its performance was fairly similar to that of RMS. One possible explanation is that -given the sparsity of the rewardall the bootstrapped networks agreed for the most part, and the algorithm simply got stuck selecting action a 1. As opposed to linear models, reacting to unusual rewards could take Bootstrapped NNs some time as good predictions could be randomly overlooked (and useful data discarded if p 1).Direct Noise Injection. When properly tuned, Parameter-Noise provided an important boost in performance across datasets over the learner that it was based on (RMS), average rank of ParamNoise-SL is 20.9 compared to RMS at 28.7 TAB0. However, we found the algorithm hard to tune and sensitive to the heuristic controlling the injected noise-level. On the synthetic Wheel problem -where exploration is necessary-both parameter-noise and RMS suffer from underexploration and perform similarly, except ParamNoise-MR which does a good job. In addition, developing an intuition for the heuristic is not straightforward as it lacks transparency and a principled grounding, and thus may require repeated access to the decision-making process for tuning. Dropout. We initially experimented with two dropout versions: fixed p = 0.5, and p = 0.8. The latter consistently delivered better , and it is the one we manually picked. The optimized versions of the algorithm provided decent improvements over its base RMS (specially Dropout-MR).In the Wheel problem, dropout performance is somewhat poor: Dropout is outperformed by RMS, while Dropout-MR offers gains with respect to all versions of RMS but it is not competitive with the best algorithms. Overall, the algorithm seems to heavily depend on its hyper-parameters (see cum-regret performance of the raw Dropout, for example). Dropout was used both for training and for decision-making; unfortunately, we did not add a baseline where dropout only applies during training. Consequently, it is not obvious how to disentangle the contribution of better training from that of better exploration. This remains as future work. Bayesian Non-parametrics. Perhaps unsurprisingly, Gaussian processes perform reasonably well on problems with little data but struggle on larger problems. While this motivated the use of sparse GP, the latter was not able to perform similarly to stronger (and definitively simpler) methods. In this work, we empirically studied the impact on performance of approximate model posteriors for decision making via Thompson Sampling in contextual bandits. We found that the most robust methods exactly measured uncertainty (possibly under the wrong model assumptions) on top of complex representations learned in parallel. More complicated approaches that learn the representation and its uncertainty together seemed to require heavier training, an important drawback in online scenarios, and exhibited stronger hyper-parameter dependence. Further exploring and developing the promising approaches is an exciting avenue for future work. Greedy NN approach, fixed learning rate (γ = 0.01). Learning rate decays, and it is reset every training period. Learning rate decays, and it is not reset at all. It starts at γ = 1. RMS Based on RMS3 net. Learning decay rate is 0.55, initial learning rate is 1.0. Trained for ts = 100, t f = 20. Based on RMS3 net. Learning decay rate is 2.5, initial learning rate is 1.0. Trained for ts = 50, t f = 20. Based on RMS3 net. Learning decay rate is 0.4, initial learning rate is 1.1. Trained for ts = 100, t f = 20. SGFS Burning = 500, learning rate γ = 0.014, EMA decay = 0.9, noise σ = 0.75. Takes each action at random with equal probability. BayesByBackprop with noise σ = 0.5. (ts = 100, first 100 times linear decay from ts = 10000). BayesByBackprop with noise σ = 0.75. (ts = 100, first 100 times linear decay from ts = 10000). BayesByBackprop with noise σ = 1.0. (ts = 100, first 100 times linear decay from ts = 10000). Bootstrapped NN Bootstrapped with q = 5 models, and p = 0.85. Based on RMS3 net. Bootstrapped NN2Bootstrapped with q = 5 models, and p = 1.0. Based on RMS3 net. Bootstrapped NN3Bootstrapped with q = 10 models, and p = 1.0. Based on RMS3 net. Dropout (RMS3) Dropout with probability p = 0.8. Based on RMS3 net. Dropout (RMS2) Dropout with probability p = 0.8. Based on RMS2 net. Greedy NN approach, fixed learning rate (γ = 0.01). Learning rate decays, and it is reset every training period. RMS2bSimilar to RMS2, but training for longer (ts = 800). Learning rate decays, and it is not reset at all. Starts at γ = 1. SGFS Burning = 500, learning rate γ = 0.014, EMA decay = 0.9, noise σ = 0.75. ConstSGD Burning = 500, EMA decay = 0.9, noise σ = 0.5. Initial noise σ = 0.01, and level = 0.01. Based on RMS3 net. Trained for longer: ts = 800. Takes each action at random with equal probability. Published as a conference paper at ICLR 2018 Table 4: Cumulative regret incurred by linear models using algorithms in Section 3 on the bandits described in Section A. Values reported are the mean over 50 independent trials with standard error of the mean. Published as a conference paper at ICLR 2018 Table 5: Simple regret incurred by linear models using algorithms in Section 3 on the bandits described in Section A. Simple regret was approximated by averaging the regret over the final 500 steps. Values reported are the mean over 50 independent trials with standard error of the mean. Statlog Covertype Financial Jester Adult Alpha Divergences 0.68 ± 0.04 0.07 ± 0.00 0.31 ± 0.00 0.00 ± 0.00 2.91 ± 0.04 0.75 ± 0.00 Alpha Divergences FORMULA1 1.50 ± 0.05 0.08 ± 0.00 0.31 ± 0.00 0.00 ± 0.00 2.98 ± 0.03 0.75 ± 0.00 Alpha Divergences FORMULA2 1.51 ± 0.05 0.13 ± 0.00 0.32 ± 0.00 0.00 ± 0.00 3.42 ± 0.05 0.77 ± 0.00 Alpha Divergences1.50 ± 0.05 0.08 ± 0.00 0.31 ± 0.00 0.00 ± 0.00 2. 0.07 ± 0.01 0.07 ± 0.00 0.29 ± 0.00 0.01 ± 0.00 2.80 ± 0.03 0.67 ± 0.00 LinGreedy (eps=0.05) 0.24 ± 0.02 0.10 ± 0.00 0.31 ± 0.00 0.06 ± 0.00 2.86 ± 0.03 0.68 ± 0.00 LinPost 0.29 ± 0.03 0.06 ± 0.00 0.28 ± 0.00 0.01 ± 0.00 2.74 ± 0.04 0.69 ± 0.00 LinfullDiagPost 4.10 ± 0.07 0.18 ± 0.00 0.63 ± 0.00 0.00 ± 0.00 2.86 ± 0.03 0.89 ± 0.00 LinfullDiagPrecPost 0.19 ± 0.02 0.05 ± 0.00 0.28 ± 0.00 0.00 ± 0.00 2.82 ± 0.03 0.67 ± 0.00 LinfullPost 0.08 ± 0.01 0.05 ± 0.00 0.28 ± 0.00 0.00 ± 0.00 2.86 ± 0.03 0.67 ± 0.00 Param-Noise 0.49 ± 0.07 0.05 ± 0.00 0.32 ± 0.00 0.01 ± 0.00 2.87 ± 0.04 0.69 ± 0.00 Param-Noise2 0.36 ± 0.05 0.05 ± 0.00 0.33 ± 0.00 0.01 ± 0.00 2.83 ± 0.04 0.69 ± 0.00 Uniform 4.88 ± 0.07 0.86 ± 0.00 0.86 ± 0.00 1.25 ± 0.02 5.03 ± 0.07 0.93 ± 0.00Published as a conference paper at ICLR 2018 Table 6: Cumulative regret incurred by models using algorithms in Section 3 on the bandits described in Section A. Values reported are the mean over 50 independent trials with standard error of the mean. Normalized with respect to the performance of Uniform. Published as a conference paper at ICLR 2018 Table 7: Simple regret incurred by models using algorithms in Section 3 on the bandits described in Section A. Simple regret was approximated by averaging the regret over the final 500 steps. Values reported are the mean over 50 independent trials with standard error of the mean. Normalized with respect to the performance of Uniform. Published as a conference paper at ICLR 2018 Table 8: Elapsed time for algorithms in Section 3 on the bandits described in Section A. Values reported are the mean over 50 independent trials with standard error of the mean. Normalized with respect to the elapsed time required by RMS (which uses t s = 100 and t f = 20).Published as a conference paper at ICLR 2018 Table 9: Cumulative regret incurred on the Wheel Bandit problem with increasing values of δ. Values reported are the mean over 50 independent trials with standard error of the mean. Normalized with respect to the performance of Uniform. We qualitatively compare plots of the sample distribution from various methods, similarly to BID19. We plot the mean and standard deviation of 100 samples drawn from each method conditioned on a small set of observations with three outputs (two are from the same underlying function and thus strongly correlated while the third (bottom) is independent). The true underlying functions are plotted in red. Mushroom. The Mushroom Dataset contains 22 attributes per mushroom, and two classes: poisonous and safe. As in BID6, we create a bandit problem where the agent must decide whether to eat or not a given mushroom. Eating a safe mushroom provides reward +5. Eating a poisonous mushroom delivers reward +5 with probability 1/2 and reward -35 otherwise. If the agent does not eat a mushroom, then the reward is 0. We set n = 50000.Statlog. The Shuttle Statlog Dataset BID2 provides the value of d = 9 indicators during a space shuttle flight, and the goal is to predict the state of the radiator subsystem of the shuttle. There are k = 7 possible states, and if the agent selects the right state, then reward 1 is generated. Otherwise, the agent obtains no reward (r = 0). The most interesting aspect of the dataset is that one action is the optimal one in 80% of the cases, and some algorithms may commit to this action instead of further exploring. In this case, n = 43500.Covertype. The Covertype Dataset BID2 classifies the cover type of northern Colorado forest areas in k = 7 classes, based on d = 54 features, including elevation, slope, aspect, and soil type. Again, the agent obtains reward 1 if the correct class is selected, and 0 otherwise. We run the bandit for n = 150000.Financial. We created the Financial Dataset by pulling the stock prices of d = 21 publicly traded companies in NYSE and Nasdaq, for the last 14 years (n = 3713). For each day, the context was the price difference between the beginning and end of the session for each stock. We synthetically created the arms, to be a linear combination of the contexts, representing k = 8 different potential portfolios. By far, this was the smallest dataset, and many algorithms over-explored at the beginning with no time to amortize their investment (Thompson Sampling does not account for the horizon).Jester. We create a recommendation system bandit problem as follows. The Jester Dataset BID15 ) provides continuous ratings in [−10, 10] for 100 jokes from 73421 users. We find a complete subset of n = 19181 users rating all 40 jokes. Following Riquelme et al. FORMULA1, we take d = 32 of the ratings as the context of the user, and k = 8 as the arms. The agent recommends one joke, and obtains the reward corresponding to the rating of the user for the selected joke. Adult. The Adult Dataset BID24 BID2 comprises personal information from the US Census Bureau database, and the standard prediction task is to determine if a person makes over $50K a year or not. However, we consider the k = 14 different occupations as feasible actions, based on d = 94 covariates (many of them binarized). As in previous datasets, the agent obtains reward 1 for making the right prediction, and 0 otherwise. We set n = 45222.Census. Dataset BID2 contains a number of personal features (age, native language, education...) which we summarize in d = 389 covariates, including binary dummy variables for categorical features. Our goal again is to predict the occupation of the individual among k = 9 classes. The agent obtains reward 1 for making the right prediction, and 0 otherwise, for each of the n = 250000 randomly selected data points. Song. The YearPredictionMSD Dataset is a subset of the Million Song Dataset BID4. The goal is to predict the year a given song was released based on d = 90 technical audio features. We divided the years in k = 10 contiguous year buckets containing the same number of songs, and provided decreasing Gaussian rewards as a function of the distance between the interval chosen by the agent and the one containing the year the song was actually released. We initially selected n = 250000 songs at random from the training set. The Statlog, Covertype, Adult, and Census datasets were tested in BID11.

Represent the TLDR for finding the corresponding scientific article

An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.