Question: Represent the article for finding a single-sentence TLDR

Question: We consider tackling a single-agent RL problem by distributing it to $n$ learners. These learners, called advisors, endeavour to solve the problem from a different focus. Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system. We show that the local planning method for the advisors is critical and that none of the ones found in the literature is flawless: the \textit{egocentric} planning overestimates values of states where the other advisors disagree, and the \textit{agnostic} planning is inefficient around danger zones. We introduce a novel approach called \textit{empathic} and discuss its theoretical aspects. We empirically examine and validate our theoretical findings on a fruit collection task. When a person faces a complex and important problem, his individual problem solving abilities might not suffice. He has to actively seek for advice around him: he might consult his relatives, browse different sources on the internet, and/or hire one or several people that are specialised in some aspects of the problem. He then aggregates the technical, ethical and emotional advice in order to build an informed plan and to hopefully make the best possible decision. A large number of papers tackle the decomposition of a single Reinforcement Learning task into several simpler ones. They generally follow a method where agents are trained independently and generally greedily to their local optimality, and are aggregated into a global policy by voting or averaging. Recent works BID12 BID30 prove their ability to solve problems that are intractable otherwise. Section 2 provides a survey of approaches and algorithms in this field. Formalised in Section 3, the Multi-Advisor RL (MAd-RL) partitions a single-agent RL into a MultiAgent RL problem BID22, under the widespread divide & conquer paradigm. Unlike Hierarchical RL BID2 BID19 BID4, this approach gives them the role of advisor: providing an aggregator with the local Q-values for all actions. The advisors are said to have a focus: reward function, state space, learning technique, etc. The MAd-RL approach allows therefore to tackle the RL task from different focuses. When a person is consulted for an advice by a enquirer, he may answer egocentrically: as if he was in charge of next actions, agnostically: anticipating any future actions equally, or empathically: by considering the next actions of the enquirer. The same approaches are modelled in the local advisors' planning methods. Section 4 shows that the egocentric planning presents the severe theoretical shortcoming of inverting a max into a max in the global Bellman equation. It leads to an overestimation of the values of states where the advisors disagree, and creates an attractor phenomenon, causing the system to remain static without any tie-breaking possibilities. It is shown on a navigation task that attractors can be avoided by lowering the discount factor γ under a given value. The agnostic planning BID30 has the drawback to be inefficient in dangerous environments, because it gets easily afraid of the controller performing a bad sequence of actions. Finally, we introduce our novel empathic planning and show that it converges to the global optimal Bellman equation when all advisors are training on the full state space.van BID29 demonstrate on a fruit collection task that a distributed architecture significantly speeds up learning and converges to a better solution than non distributed baselines. Section 5.2 extends those and empirically validates our theoretical analysis: the egocentric planning gets stuck in attractors with high γ values; with low γ values, it gets high scores but is also very unstable as soon as some noise is introduced; the agnostic planning fails at efficiently gathering the fruits near the ghosts; despite lack of convergence guarantees with partial information in advisors' state space, our novel empathic planning also achieves high scores while being robust to noise. Task decomposition -Literature features numerous ways to distribute a single-agent RL problem over several specialised advisors: state space approximation/reduction BID0, reward segmentation BID2 BID8 BID31 BID30, algorithm diversification BID32 BID14, algorithm randomization BID1 BID10, sequencing of actions BID27, or factorisation of actions BID15. In this paper, we mainly focus on reward segmentation and state space reduction but the findings are applicable to any family of advisors. BID23 are the first to propose to merge Markov decision Processes through their value functions. It makes the following strong assumptions: positive rewards, model-based RL, and local optimality is supposed to be known. Finally, the algorithm simply accompanies a classical RL algorithm by pruning actions that are known to be suboptimal. BID24 propose to use a local SARSA online learning algorithm for training the advisors, but they elude the fact that the online policy cannot be locally accurately estimated with partial state space, and that this endangers the convergence properties. BID21 study more in depth the theoretical guaranties of convergence to optimality with the local Q-learning, and the local SARSA algorithms. However, their work is limited in the fact that they do not allow the local advisors to be trained on local state space. van BID30 relax this assumption at the expense of optimality guarantees and beat one of the hardest Atari games: Ms. Pac-Man, by decomposing the task into hundreds of subtasks that are trained in parallel. MAd-RL can also be interpreted as a generalisation of Ensemble Learning BID5 for RL. As such, BID25 use a boosting algorithm in a RL framework, but the boosting is performed upon policies, not RL algorithms. In this sense, this article can be seen as a precursor to the policy reuse algorithm BID7 rather than a multi-advisor framework. BID32 combine five online RL algorithms on several simple RL problems and show that some mixture models of the five experts performs generally better than any single one alone. Each algorithm tackles the whole task. Their algorithms were off-policy, on-policy, actor-critics, etc. BID6 continue this effort in a very specific setting where actions are explicit and deterministic transitions. We show in Section 4 that the planning method choice is critical and that some recommendations can be made in accordance to the task definition. In BID11, while all advisors are trained on different reward functions, these are potential based reward shaping variants of the same reward function. They are therefore embedding the same goals. As a consequence, it can be related to a bagging procedure. The advisors recommendation are then aggregated under the HORDE architecture BID28, with egocentric planning. Two aggregator functions are tried out: majority voting and ranked voting. BID14 follow a different approach in which, instead of boosting the weak advisors performances by aggregating their recommendation, they select the best advisor. This approach is beneficial for staggered learning, or when one or several advisors may not find good policies, but not for variance reduction brought by the committee, and it does not apply to compositional RL.The UNREAL architecture BID12 improves the state-of-the art on Atari and Labyrinth domains by training their deep network on auxiliary tasks. They do it in an unsupervised manner and do not consider each learner as a direct contributor to the main task. The bootstrapped DQN architecture BID17 also exploits the idea of multiplying the Q-value estimations to favour deep exploration. As a , UNREAL and bootstrapped DQN do not allow to break down a task into smaller, tractable pieces. As a summary, a large variety of papers are published on these subjects, differing by the way they factorise the task into subtasks. Theoretical obstacles are identified in BID23 and BID21, but their analysis does not go further than the non-optimality observation in the general case. In this article, we accept the non-optimality of the approach, because it naturally comes from the simplification of the task brought by the decomposition and we analyse the pros and cons of each planning methods encountered in the literature. But first, Section 3 lays the theoretical foundation for Multi-Advisor RL.Domains -Related works apply their distributed models to diverse domains: racing BID21, scheduling BID21, dialogue BID14, and fruit collection BID23 BID24 BID29 b). Pac-Boy is yellow, the corridors are in black, the walls in grey, the fruits are the white dots and the ghosts are in red. The fruit collection task being at the centre of attention, it is natural that we empirically validate our theoretical findings on this domain: the Pac-Boy game (see FIG0, borrowed from van BID29 . Pac-Boy navigates in a 11x11 maze with a total of 76 possible positions and 4 possible actions in each state: A = {N, W, S, E}, respectively for North, West, South and East. Bumping into a wall simply causes the player not to move without penalty. Since Pac-Boy always starts in the same position, there are 75 potential fruit positions. The fruit distribution is randomised: at the start of each new episode, there is a 50% probability for each position to have a fruit. A game lasts until the last fruit has been eaten, or after the 300 th time step. During an episode, fruits remain fixed until they get eaten by Pac-Boy. Two randomly-moving ghosts are preventing Pac-Boy from eating all the fruits. The state of the game consists of the positions of PacBoy, fruits, and ghosts: 76 × 2 75 × 76 2 ≈ 10 28 states. Hence, no global representation system can be implemented without using function approximation. Pac-Boy gets a +1 reward for every eaten fruit and a −10 penalty when it is touched by a ghost. Markov Decision Process -The Reinforcement Learning (RL) framework is formalised as a Markov Decision Process (MDP). An MDP is a tuple X, A, P, R, γ where X is the state space, A is the action space, P: X × A → X is the Markovian transition stochastic function, R: X × A → R is the immediate reward stochastic function, and γ is the discount factor. A trajectory x(t), a(t), x(t + 1), r(t) t∈ 0,T −1 is the projection into the MDP of the task episode. The goal is to generate trajectories with high discounted cumulative reward, also called more succinctly return: DISPLAYFORM0 To do so, one needs to find a policy π: DISPLAYFORM1 Advisor Advisor... Figure 2: The MAd-RL architectureMAd-RL structure -This section defines the Multi-Advisor RL (MAd-RL) framework for solving a single-agent RL problem. The n advisors are regarded as specialised, possibly weak, learners that are concerned with a sub part of the problem. Then, an aggregator is responsible for merging the advisors' recommendations into a global policy. The overall architecture is illustrated in Figure 2. At each time step, an advisor j sends to the aggregator its local Q-values for all actions in the current state. Aggregating advisors' recommendations -In Figure 2, the f function's role is to aggregate the advisors' recommendations into a policy. More formally, the aggregator is defined with f: R n×|A| → A, which maps the received q j = Q j (x j, •) values into an action of A. This article focuses on the analysis of the way the local Q j -functions are computed. From the values q j, one can design a f function that implements any aggregator function encountered in the Ensemble methods literature BID5: voting schemes BID9, Boltzmann policy mixtures BID32 and of course linear value-function combinations BID25 BID21 BID30. For the further analysis, we restrict ourselves to the linear decomposition of the rewards: R(x, a) = j w j R j (x j, a), which implies the same decomposition of return if they share the same γ. The advisor's state representation may be locally defined by φ j: X → X j, and its local state is denoted by x j = φ j (x) ∈ X j. We define the aggregator function f Σ (x) as being greedy over the Q j -functions aggregation Q Σ (x, a): DISPLAYFORM0 We recall hereunder the main theoretical of BID30: a theorem ensuring, under conditions, that the advisors' training eventually converges. Note that by assigning a stationary behaviour to each of the advisors, the sequence of random variables X 0, X 1, X 2,..., with X t ∈ X is a Markov chain. For later analysis, we assume the following. Assumption 1. All the advisors' environments are Markov: DISPLAYFORM1 Theorem 1 (van BID30). Under Assumption 1 and given any fixed aggregator, global convergence occurs if all advisors use off-policy algorithms that converge in the single-agent setting. Although Theorem 1 guarantees convergence, it does not guarantee the optimality of the converged solution. Moreover, this fixed point only depends on each advisor model and on their planning methods (see Section 4), but not on the particular optimisation algorithms that are used by them. This section present three planning methods at the advisor's level. They differ in the policy they evaluate: egocentric planning evaluates the local greedy policy, agnostic planning evaluates the random policy, and empathic planning evaluates the aggregator's greedy policy. The most common approach in the literature is to learn off-policy by bootstrapping on the locally greedy action: the advisor evaluates the local greedy policy. This planning, referred to in this paper as egocentric, has already been employed in BID23, BID21, BID11, and van BID29. Theorem 1 guarantees for each advisor j the convergence to the local optimal value function, denoted by Q ego j, which satisfies the Bellman optimality equation: DISPLAYFORM0 where the local immediate reward r j is sampled according to R j (x j, a), and the next local state x j is sampled according to P j (x j, a). In the aggregator global view, we get: By construction, r = j w j r j, and therefore we get: DISPLAYFORM1 DISPLAYFORM2 Egocentric planning suffers from an inversion between the max and operators and, as a consequence, it overestimate the state-action values when the advisors disagree on the optimal action. This flaw has critical consequences in practice: it creates attractor situations. Before we define and study them formally, let us explain attractors with an illustrative example based on the simple MDP depicted in FIG1. In initial state x 0, the system has three possible actions: stay put (action a 0), perform advisor 1's goal (action a 1), or perform advisor 2's goal (action a 2). Once achieving a goal, the trajectory ends. The Q-values for each action are easy to compute: DISPLAYFORM3 As a consequence, if γ > r 1 /(r 1 + r 2) and γ > r 2 /(r 1 + r 2), the local egocentric planning commands to execute action a 0 sine die. This may have some apparent similarity with the Buridan's ass paradox BID33 BID20: a donkey is equally thirsty and hungry and cannot decide to eat or to drink and dies of its inability to make a decision. The determinism of judgement is identified as the source of the problem in antic philosophy. Nevertheless, the egocentric sub-optimality does not come from actions that are equally good, nor from the determinism of the policy, since adding randomness to the system will not help. Let us define more generally the concept of attractors. Definition 1. An attractor x is a state where the following strict inequality holds: DISPLAYFORM4 Theorem 2. State x is attractor, if and only if the optimal egocentric policy is to stay in x if possible. Note that there is no condition in Theorem 2 (proved in appendix, Section A) on the existence of actions allowing the system to be actually static. Indeed, the system might be stuck in an attractor set, keep moving, but opt to never achieve its goals. To understand how this may happen, just replace state x 0 in FIG1 with an attractor set of similar states: where action a 0 performs a random transition in the attractor set, and actions a 1 and a 2 respectively achieve tasks of advisors 1 and 2. Also, it may happen that an attractor set is escapable by the lack of actions keeping the system in an attractor set. For instance, in FIG1, if action a 0 is not available, x 0 remains an attractor, but an unstable one. Definition 2. An advisor j is said to be progressive if the following condition is satisfied: DISPLAYFORM5 The intuition behind the progressive property is that no action is worse than losing one turn to do nothing. In other words, only progress can be made towards this task, and therefore non-progressing actions are regarded by this advisor as the worst possible ones. Theorem 3. If all the advisors are progressive, there cannot be any attractor. The condition stated in Theorem 3 (proved in appendix, Section A) is very restrictive. Still, there exist some RL problems where Theorem 3 can be applied, such as resource scheduling where each advisor is responsible for the progression of a given task. Note that a MAd-RL setting without any attractors does not guarantee optimality for the egocentric planning. Most of RL problems do not fall into this category. Theorem 3 neither applies to RL problems with states that terminate the trajectory while some goals are still incomplete, nor to navigation tasks: when the system goes into a direction that is opposite to some goal, it gets into a state that is worse than staying in the same position. Navigation problem attractors -We consider the three-fruit attractor illustrated in FIG2: moving towards a fruit, makes it closer to one of the fruits, but further from the two other fruits (diagonal moves are not allowed). The expression each action Q-value is as follows: DISPLAYFORM6 As a , the aggregator would opt to go South and hit the wall indefinitely. More generally in a deterministic 1 task where each action a in a state x can be cancelled by a new action a -1 x, it can be shown that the condition on γ is a function of the size of the action set A. Theorem 4 is proved in Section A of the appendix. Theorem 4. State x ∈ X is guaranteed not to be an attractor if ∀a ∈ A, ∃a -1x ∈ A, such that P (P (x, a), a The agnostic planning does not make any prior on future actions and therefore evaluates the random policy. Once again, Theorem 1 guarantees the convergence of the local optimisation process to its local optimal value, denoted by Q agn j, which satisfies the following Bellman equation: DISPLAYFORM0 Local agnostic planning is equivalent to the global agnostic planning. Additionally, as opposed to the egocentric case, r.h.s. of the above equation does not suffer from max-inversion. It then follows that no attractor are present in agnostic planning. Nevertheless, acting greedily with respect to Q agn Σ (x, a) only guarantees to be better than a random policy and in general may be far from being optimal. Still, agnostic planning has proved its usefulness on Ms. Pac-Man (van b). A novel approach, inspired from the online algorithm found in BID24; BID21 is to locally predict the aggregator's policy. In this method, referred to as empathic, the aggregator is in control, and the advisors are evaluating the current aggregator's greedy policy f with respect to their local focus. More formally, the local Bellman equilibrium equation is the following one: Q ap j (x j, a) = E r j + γQ ap j (x j, f Σ (x)). Theorem 5. Assuming that all advisors are defined on the full state space, MAd-RL with empathic planning converges to the global optimal policy. Proof. DISPLAYFORM0 Σ is the unique solution to the global Bellman optimality equation, and therefore equals the optimal value function, quod erat demonstrandum. However, most MAd-RL settings involve taking advantage of state space reduction to speed up learning, and in this case, there is no guarantee of convergence because function f Σ (x) can only be approximated from the local state space scope. As a the local estimatef j (x) is used instead of f Σ (x) and the reconstruction of max a ∈A Q ap Σ (x, a) is not longer possible in the global Bellman equation: DISPLAYFORM1 For this experiment, we intend to show that the value function is easier to learn with the MAd-RL architecture. We consider a fruit collection task where the agent has to navigate through a 5 × 5 grid and receives a +1 reward when visiting a fruit cell (5 are randomly placed at the beginning of each episode). A deep neural network (DNN) is fitted to the ground-truth value function V π * γ for various objective functions: TSP is the optimal number of turns to gather all the fruits, RL is the optimal return, and egocentric is the optimal MAd-RL return. This learning problem is fully supervised on 1000 samples, allowing us to show how fast a DNN can capture V π * γ while ignoring the burden of finding the optimal policy and estimating its value functions through TD-backups or value iteration. To evaluate the DNN's performance, actions are selected greedily by moving the agent up, down, left, or right to the neighbouring grid cell of highest value. Section B.1 of the appendix gives the details. FIG6 displays the performance of the theoretical optimal policy for each objective function in dashed lines. Here TSP and RL targets largely surpass MAd-RL one. But FIG6 also displays the performances of the networks trained on the limited data of 1000 samples, for which the are completely different. The TSP objective target is the hardest to train on. The RL objective target follows as the second hardest to train on. The egocentric planning MAd-RL objective is easier to train on, even without any state space reduction, or even without any reward/return decomposition (summed version). Additionally, if the target value is decomposed (vector version), the training is further accelerated. Finally, we found that the MAd-RL performance tends to dramatically decrease when γ gets close to 1, because of attractors' presence. We consider this small experiment to show that the complexity of objective function is critical and that decomposing it in the fashion of MAd-RL may make it simpler and therefore easier to train, even without any state space reduction. In this section, we empirically validate the findings of Section 4 in the Pac-Boy domain, presented in Section 2. The MAd-RL settings are associating one advisor per potential fruit location. The local state space consists in the agent position and the existence -or not-of the fruit. Six different settings are compared: the two baselines linear Q-learning and DQN-clipped, and four MAd-RL settings: egocentric with γ = 0.4, egocentric with γ = 0.9, agnostic with γ = 0.9, and empathic with γ = 0.9. The implementation and experimentation details are available in the appendix, at Section B.2.We provide links to 5 video files (click on the blue links) representing a trajectory generated at the 50 th epoch for various settings. egocentric-γ = 0.4 adopts a near optimal policy coming close to the ghosts without taking any risk. The fruit collection problem is similar to the travelling salesman problem, which is known to be NP-complete BID18. However, the suboptimal small-γ policy consisting of moving towards the closest fruits is in fact a near optimal one. Regarding the ghost avoidance, egocentric with small γ gets an advantage over other settings: the local optimisation guarantees a perfect control of the system near the ghosts. The most interesting outcome is the presence of the attractor phenomenon in egocentric-γ = 0.9: Pac-Boy goes straight to the centre area of the grid and does not move until a ghost comes too close, which it still knows to avoid perfectly. This is the empirical confirmation that the attractors, studied in Section 4.1, present a real practical issue. empathic is almost as good as egocentric-γ = 0.4. agnostic proves to be unable to reliably finish the last fruits because it is overwhelmed by the fear of the ghosts, even when they are still far away. This feature of the agnostic planning led van BID30 to use a dynamic normalisation depending on the number of fruits left on the board. Finally, we observe that DQN-clipped also struggles to eat the last fruits. The quantitative analysis displayed in FIG6 confirms our qualitative video-based impressions. egocentric-γ = 0.9 barely performs better than linear Q-learning, DQN-clipped is still far from the optimal performance, and gets hit by ghosts from time to time. agnostic is closer but only rarely eats all the fruits. Finally, egocentric-γ = 0.4 and empathic are near-optimal. Only egocentric-γ = 0.4 trains a bit faster, and tends to finish the game 20 turns faster too (not shown).Results with noisy rewards -Using a very small γ may distort the objective function and perhaps even more importantly the reward signal diminishes exponentially as a function of the distance to the number of turns network trained on vector egocentric objective network trained on summed egocentric objective network trained on RL objective network trained on TSP objective optimal policy given egocentric objective optimal policy given RL objective optimal policy given TSP objective (a) Value function training. goal, which might have critical consequences in noisy environments, hence the following experiment: several levels of Gaussian centred white noise η σ with standard deviation σ ∈ {0.01, 0.1} have been applied to the reward signal: at each turn, each advisor now receivesr j = r j + η σ instead. Since the noise is centred and white, the ground truth Q-functions remain the same, but their estimators obtained during sampling is corrupted by noise variance. Empirical displayed in FIG6 shows that the empathic planning performs better than the egocentric one even under noise with a 100 times larger variance. Indeed, because of the noise, the fruit advisors are only able to consistently perceive the fruits that are in a radius dependent on γ and σ. The egocentric planning, incompatible with high γ values, is therefore myopic and cannot perceive distant fruits. The same kind of limitations are expected to be encountered for small γ values when the local advisors rely on state approximations, and/or when the transitions are stochastic. This also supports the superiority of the empathic planning in the general case. This article presented MAd-RL, a common ground for the many recent and successful works decomposing a single-agent RL problem into simpler problems tackled by independent learners. It focuses more specifically on the local planning performed by the advisors. Three of them -two found in the literature and one novel -are discussed, analysed and empirically compared: egocentric, agnostic, and empathic. The lessons to be learnt from the article are the following ones. The egocentric planning has convergence guarantees but overestimates the values of states where the advisors disagree. As a consequence, it suffers from attractors: states where the no-op action is preferred to actions making progress on a subset of subtasks. Some domains, such as resource scheduling, are identified as attractor-free, and some other domains, such as navigation, are set conditions on γ to guarantee the absence of attractor. It is necessary to recall that an attractor-free setting means that the system will continue making progress towards goals as long as there are any opportunity to do so, not that the egocentric MAd-RL system will converge to the optimal solution. The agnostic planning also has convergence guarantees, and the local agnostic planning is equivalent to the global agnostic planning. However, it may converge to bad solutions. For instance, in dangerous environments, it considers all actions equally likely, it favours staying away from situation where a random sequence of actions has a significant chance of ending bad: crossing a bridge would be avoided. Still, the agnostic planning simplicity enables the use of general value functions BID28 BID30.The empathic planning optimises the system according to the global Bellman optimality equation, but without any guarantee of convergence, if the advisor state space is smaller than the global state. In our experiments, we never encountered a case where the convergence was not obtained, and on the Pac-Boy domain, it robustly learns a near optimal policy after only 10 epochs. It can also be safely applied to Ensemble RL tasks where all learners are given the full state space. Theorem 1 BID30 ). Under Assumption 1 and given any fixed aggregator, global convergence occurs if all advisors use off-policy algorithms that converge in the single-agent setting. Proof. Each advisor can be seen as an independent learner training from trajectories controlled by an arbitrary behavioural policy. If Assumption 1 holds, each advisor's environment is Markov and off-policy algorithms can be applied with convergence guarantees. Theorem 2. State x is attractor, if and only if the optimal egocentric policy is to stay in x if possible. Proof. The logical biconditional will be demonstrated by successively proving the two converse conditionals. First, the sufficient condition: let us assume that state x is an attractor. By Definition 1, if state x is an attractor, we have: DISPLAYFORM0 Let a 0 denote the potential action to stay in x, and consider the MDP augmented with a 0 in x. Then, we have: DISPLAYFORM1 which proves that a 0, if possible, will be preferred to any other action a ∈ A.Second, the reciprocal condition: let us assume that, in state x, action a 0 would be preferred by an optimal policy under the egocentric planning. Then: DISPLAYFORM2 which proves that x is an attractor. Theorem 3. If all the advisors are progressive, there cannot be any attractor. Proof. Let sum Definition 2 over advisors: DISPLAYFORM3 which proves the theorem. Theorem 4. State x ∈ X is guaranteed not to be an attractor if:• ∀a ∈ A, ∃a -1x ∈ A, such that P (P (x, a), a -1 DISPLAYFORM4 Under review as a conference paper at ICLR 2018Proof. Let us denote J x a as the set of advisors for which action a is optimal in state x. Q ego a (x) is defined as the sum of perceived value of performing a in state x by the advisors that would choose it: DISPLAYFORM5 Let a + be the action that maximises this Q ego a (x) function: DISPLAYFORM6 We now consider the left hand side of the inequality characterising the attractors in Definition 1: DISPLAYFORM7 Since R(x, a +) ≥ 0, and since the a maximising Q ego j (x j, a) is at least as good as the cancelling action (a +)-1x, we can follow with: DISPLAYFORM8 By comparing this last with the right hand side of Definition 1, the condition for x not being an attractor becomes: DISPLAYFORM9 It follows directly from the inequality Q Similarly to the , we incorporate the location of the fruits into the state representation by using a 50 dimensional bit vector, where the first 25 entries are used for fruit positions, and the last 25 entries are used for the agent's position. The DNN feeds this bit-vector as the input layer into two dense hidden layers with 100 and then 50 units. The output is a single linear head representing the state-value, or a multiple in the case of the vector MAd-RL target. In order to assess the value function complexity, we train for each discount factor setting a DNN of fixed size on 1000 random states with their ground truth values. Each DNN is trained over 500 epochs using the Adam optimizer with default parameters. Four different objective function targets are considered:• The TSP objective function target is the natural objective function, as defined by the Travelling Salesman Problem: the number of turns to gather all the fruits: DISPLAYFORM0 where k is the number of fruits remaining in statex, where Σ k is the ensemble of all permutations of integers between 1 and k, where σ is one of those permutations, where x 0 is the position of the agent in x, where x i for 1 ≤ i ≤ k is the position of fruit with index i, where d(x i, x j) is the distance (||·|| 1 in our gridworld) between positions x i and x j.• The RL objective function target is the objective function defined for an RL setting, which depends on the discount factor γ: DISPLAYFORM1 with the same notations as for TSP.• The summed egocentric planning MAd-RL objective function target does not involve the search into the set of permutations and can be considered simpler to this extent: DISPLAYFORM2 with the same notations as for TSP.• The vector egocentric planning MAd-RL objective function target is the same as the summed one, except that the target is now a vector, separated into as many channels as potential fruit position: x0,xi) if there is a fruit in x i, 0 otherwise. DISPLAYFORM3 MAd-RL Setup -Each advisor is responsible for a specific source of reward (or penalty). More precisely, we assign an advisor to each possible fruit location. This advisor sees a +1 reward only if a fruit at its assigned position gets eaten. Its state space consists of Pac-Boy's position, ing in 76 states. In addition, we assign an advisor to each ghost. This advisor receives a -10 reward if Pac-Boy bumps into its assigned ghost. Its state space consists of Pac-Boy's and ghost's positions, ing in 76 2 states. A fruit advisor is only active when there is a fruit at its assigned position. Because there are on average 37.5 fruits, the average number of advisors running at the beginning of each episode is 39.5. Each fruit advisor is set inactive when its fruit is eaten. The learning was performed through Temporal Difference updates. Due to the small state spaces for the advisors, we can use a tabular representation. We train all learners in parallel with off-policy learning, with Bellman residuals computed as presented in Section 4 and a constant α = 0.1 parameter. The aggregator function sums the Q-values for each action a ∈ A: Q Σ (x, a):= j Q j (x j, a), and uses -greedy action selection with respect to these summed values. Because ghost agents have exactly identical MDP, we also benefit from direct knowledge transfer by sharing their Q-tables. One can notice that Assumption 1 holds in this setting and that, as a consequence, Theorem 1 applies for the egocentric and agnostic planning methods. Theorem 4 determines sufficient conditions for not having any attractor in the MDP. In the Pac-Boy domain, the cancelling action condition is satisfied for every x ∈ X. As for the γ condition, it is not only sufficient but also necessary, since being surrounded by goals of equal value is an attractor if γ > 1/3. In practice, an attractor becomes stable only when there is an action enabling it to remain in the attraction set. Thus, the condition for not being stuck in an attractor set can be relaxed to γ ≤ 1/(|A|−2). Hence, the of γ > 1/2 in the example illustrated by FIG2.Baselines -The first baseline is the standard DQN algorithm BID16 with reward clipping (referred to as DQN-clipped). Its input is a 4-channel binary image with the following features: the walls, the ghosts, the fruits, or Pac-Boy. The second baseline is a system that uses the exact same input features as the MAd-RL model. Specifically, the state of each advisor of the MAd-RL model is encoded with a one-hot vector and all these vectors are concatenated, ing in a sparse binary feature vector of size 17, 252. This vector is used for linear function approximation with Q-learning. We refer to this setting with linear Q-learning. We also tried to train deep architectures from these features with no success. Experimental setting -Time scale is divided into 50 epochs lasting 20,000 transitions each. At the end of each epoch an evaluation phase is launched for 80 games. The theoretical expected maximum score is 37.5 and the random policy average score is around -80.Explicit links to the videos (www.streamable.com website was used to ensure anonymity, if accepted the videos will be linked to a more sustainable website):• egocentric-γ = 0.4: https://streamable.com/6tian• egocentric-γ = 0.9: https://streamable.com/sgjkq• empathic: https://streamable.com/h6gey• agnostic: https://streamable.com/grswh• DQN-clipped: https://streamable.com/emh6y

We consider tackling a single-agent RL problem by distributing it to $n$ learners.

represent the tldr for finding the corresponding scientific article.

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.


Question: We present a hybrid framework that leverages the trade-off between temporal and frequency precision in audio representations to improve the performance of speech enhancement task. We first show that conventional approaches using specific representations such as raw-audio and spectrograms are each effective at targeting different types of noise. By integrating both approaches, our model can learn multi-scale and multi-domain features, effectively removing noise existing on different regions on the time-frequency space in a complementary way. Experimental show that the proposed hybrid model yields better performance and robustness than using each model individually. We first describe the objective function and the selected modules that have been reported to show 36 competitive performance using either raw-audio or spectrogram input. Selected models are 37 each used later as components of our proposed hybrid model. We employ the energy-conserving loss function proposed in which simultaneously considers 40 speech and noise signals. Let the noisy input x consist of clean speech s and noise n. The estimated 41 speech by the model is referred to asŝ. Then, our objective function is defined as follows: L(x, s, n,ŝ) = s −ŝ 1 + n −n 1,wheren = x −ŝ represents the estimated noise signal and · 1 denotes 1 norm. We construct the time domain network based on TasNet which employs one-dimensional dilated 45 convolution to handle long time sequences of raw-audio. TasNet has shown competitive sample quality 46 for speech source separation, which is a similar task to speech enhancement. In our experiments, we used a reduced version of TasNet. With a slight abuse of notation, we refer to the network as noise from the time-frequency space. We hybridize both time and T-F domain networks in a cascaded way (Fig. 1 Figure 1 : A schematic illustration of the hybrid system (MDPhD). Note that the network of the same domain (same color) shares the parameters. For the time-frequency (T-F) domain network, we convert the time-domain input to a spectrogram using the short time Fourier transform (STFT), whose output is converted back to a waveform using the inverse short time Fourier transform (iSTFT).The final objective of the hybrid model with auxiliary loss becomes DISPLAYFORM0 where θ denotes the network parameter. that of U-Net with doubled parameter size (TAB3). Note that, however, TasNet fails to remove high 88 frequency noise, which is supposedly hard to capture in the time domain FIG2. the U → D model shares the weakness of U-Net and vice versa. We conjecture that this happens Using the test dataset, we compared our to recent studies of speech enhancement field. Our model showed the best performance quantitatively and qualitatively among the others under TAB4: Comparison with other methods. The predicted rating of speech distortion (CSIG), distortion (CBAK) and overall quality (COVL) are reported (from 1 to 5, higher is better). PESQ (from -0.5 to 4.5, higher is better) stands for perceptual evaluation of speech quality and SSNR (higher is better) is segmental SNR. The best for each measure is given in bold style. Table 3: SNR evaluation of models with various objective functions. D and U denote the TasNet (reduced) using one-dimensional dilated convolution and U-Net, respectively. The type of objective functions are noted next to the model name. 1 represents our baseline objective function. 2 represents an objective function that substitutes the 1 term of equation FORMULA0 In this section, we present the detailed configuration of the models we used. In the following figures, Figure 3: U-Net (1.5M) architecture. 2D Conv means a two-dimensional convolution block consisting of a two-dimensional convolution operation with filter size F (height, width), stride size S (height, width) and output channel size C followed by batch renormalization and leaky-RELU activation function. 2D t-Conv means a two-dimensional transposed convolution block. Our baseline models used in experiments process the log-magnitude of the input spectrogram.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A hybrid model utilizing both raw-audio and spectrogram information for speech enhancement tasks.

Represent the TLDR for finding the corresponding scientific article

A new large batch training algorithm  based on Layer-wise Adaptive Rate Scaling (LARS); using LARS, we scaled AlexNet  and ResNet-50 to a batch of 16K.


Question: Represent the scientific article to find a short TLDR

Question: Consistently checking the statistical significance of experimental is the first mandatory step towards reproducible science. This paper presents a hitchhiker's guide to rigorous comparisons of reinforcement learning algorithms. After introducing the concepts of statistical testing, we review the relevant statistical tests and compare them empirically in terms of false positive rate and statistical power as a function of the sample size (number of seeds) and effect size. We further investigate the robustness of these tests to violations of the most common hypotheses (normal distributions, same distributions, equal variances). Beside simulations, we compare empirical distributions obtained by running Soft-Actor Critic and Twin-Delayed Deep Deterministic Policy Gradient on Half-Cheetah. We conclude by providing guidelines and code to perform rigorous comparisons of RL algorithm performances. Reproducibility in Machine Learning and Reinforcement Learning in particular (RL) has become a serious issue in the recent years. As pointed out in Islam et al. BID0 and Henderson et al. BID1, reproducing the of an RL paper can turn out to be much more complicated than expected. In a thorough investigation, Henderson et al. BID1 showed it can be caused by differences in codebases, hyperparameters (e.g. size of the network, activation functions) or the number of random seeds used by the original study. Henderson et al. BID1 states the obvious: the claim that an algorithm performs better than another should be supported by evidence, which requires the use of statistical tests. Building on these observations, this paper presents a hitchhiker's guide for statistical comparisons of RL algorithms. The performances of RL algorithm have specific characteristics (they are independent of each other, they are not paired between algorithms etc.). This paper reviews some statistical tests relevant in that context and compares them in terms of false positive rate and statistical power. Beside simulations, it compares empirical distributions obtained by running Soft-Actor Critic (SAC) BID2 and Twin-Delayed DDPG (TD3) BID3 on Half-Cheetah BID4. We finally provide guidelines to perform robust difference testing in the context of RL. A repository containing the raw and the code to reproduce all experiments is available at https://github.com/ccolas/rl_stats. In this paper, we consider the problem of conducting meaningful comparisons of Algorithm 1 and Algorithm 2. Because the seed of the random generator is different for each run BID1, two runs of a same algorithm yield different measures of performance. An algorithm performance can therefore be modeled as a random variable X, characterized by a distribution. Measuring the performance x at the end of a particular run is equivalent to measuring a realization of that random variable. Repeating this N times, we obtain a sample x = (x 1, ..., x N) of size N. To compare RL algorithms on the basis of their performances, we focus on the comparisons of the central tendencies (µ 1, µ 2): the means or the medians of the associated random variables X 1, X 2. BID2 Unfortunately, we cannot know µ 1, µ 2 exactly. Given a sample x i of X i, we can estimate µ i by the empirical mean: DISPLAYFORM0 (resp. the empirical median). However, comparing central performances does not simply boil down to the comparison of their estimates. As an illustration, FIG0 shows two normal distributions describing the distributions of two algorithm performances X 1 and X 2. Two samples of sample size N = 3 are collected. In this example, we have µ 1 < µ 2 but x 1 > x 2. The rest of this text uses central performance to refer to either the mean or the median of the performance distribution i. It is noted µ i while its empirical estimate is noted x i. The distinction is made where necessary. Statistical difference testing. Statistical difference testing offers a principled way to compare the central performances of two algorithms. It defines two hypothesis: 1) the null hypothesis H 0: ∆µ = µ 1 −µ 2 = 0 and 2) the alternative hypothesis H a: |∆µ| > 0. When performing a test, one initially assumes the null hypothesis to be true. After having observed (x 1, x 2), statistical tests usually estimate the probability to observe two samples whose empirical central difference is at least as extreme as the observed one (|∆x| = |x 1 −x 2 |) under H 0 (e.g. given ∆µ = 0). This probability is called the p-value. If the p-value is very low, the test rejects H 0 and concludes that a true underlying difference (H a) is likely. When the p-value is high, the test does not have enough evidence to conclude. This could be due to the lack of true difference, or to the lack of statistical power (too few measurements given how noisy they are). The significance level α (usually ≤ 0.05) draws the line between rejection and conservation of H 0: if p-value < α, H 0 is rejected. Further experiments demonstrate it is not always the case, which is why we prefer to note the false positive rate α *. False negatives occur when the statistical test fails to recognize a true difference in the central performances. This depends on the size of the underlying difference: the larger the difference, the lower the risk of false negative. The false negative rate is noted β *.Trade-off between false positive and statistical power. Ideally, we would like to set α = 0 to ensure the lowest possible false positive rate α *. However, decreasing the confidence level makes the statistical test more conservative. The test requires even bigger empirical differences ∆x to reject H 0, which decreases the probability of true positive. This probability of true positive 1−β * is called the statistical power of a test. It is the probability to reject H 0 when H a holds. It is directly impacted by the effect size: the larger the effect size, the easier it is to detect (larger statistical power). It is also a direct function of the sample size: larger samples bring more evidence to support the rejection of H 0. Generally, the sample size is chosen so as to obtain a theoretical statistical power of 1−β * = 0.8. Different tests have different statistical powers depending on the assumptions they make, whether they are met, how the p-value is derived etc. Parametric vs. non-parametric. Parametric tests usually compare the means of two distributions by making assumptions on the distributions of the two algorithms' performances. Non-parametric tests on the other hand usually compare the medians and do not require assumptions on the type of distributions. Non-parametric tests are often recommended when one wants to compare median rather than means, when the data is skewed or when the sample size is small. Section 4.2 shows that these recommendations are not always justified. Test statistic. Statistical tests usually use a test statistic. It is a numerical quantity computed from the samples that summarizes the data. In the t-test for instance, the statistic t α is computed as t α = |∆x|/σ pool, where σ pool is the pooled standard deviation (σ pool = (σ 2 1 + σ 2 2)/2). Under the t-test assumptions, this statistic follows the analytic Student's distribution with density function f S (t). The probability to obtain a difference more important than the sample difference ∆x (p-value) can be rewritten p-value = P (|t| > t α) and can be computed as the area under f S (t) such that |t| > t α.Relative effect size. The relative effect size is the absolute effect size |∆µ|, scaled by the pooled standard deviation σ pool, such that = |∆µ|/σ pool. The relative effect size is independent of the spread of the considered distributions.3 Statistical Tests for RL Each test makes some assumptions (e.g. about the nature of the performance distributions, their variances, the sample sizes etc.). In the context of RL, some assumptions are reasonable while others are not. It is reasonable to assume that RL performances are measured at random and independently from one another. The samples are not paired, and here we assume they have the same size. BID3 Other common assumptions might be discussed:• Normal distributions of performances: this might not be the case (skewed distributions, bimodal distributions, truncated distributions).• Continuous performances: the support of the performance distribution might be bounded:e.g. in the Fetch environments of Gym BID4, the performance is a success rate in.• Known standard deviations: this is not the case in RL.• Equal standard deviations: this is often not the case (see BID1). This section briefly presents various statistical tests relevant to the comparison of RL performances. It focuses on the underlying assumptions BID5 and provides the corresponding implementation from the Python Scipy library when available. Further details can be found in any statistical textbook. Contrary to Henderson et al. BID1, we do not recommend using the Kolmogorov-Smirnov test as it tests for the equality of the two distributions and does not test for a difference in their central tendencies BID6.T-test. This parametric test compares the means of two distributions and assumes the two distributions have equal variances BID7. If this variance is known, a more powerful test is available: the Z-test for two population means. The test is accurate when the two distributions are normal, it gives an approximate guide otherwise. Implementation: scipy.stats.ttest_ind(x1, x2, equal_var=True).Welch's t-test. It is a t-test where the assumption of equal variances is relaxed BID8. Implementation: scipy.stats.ttest_ind(x1, x2, equal_var=False).Wilcoxon Mann-Whitney rank sum test. This non-parametric test compares the median of two distributions. It does not make assumptions about the type of distributions but assumes they are continuous and have the same shape and spread BID9. Implementation: scipy.stats.mannwhitneyu(x1, x2, alternative='two-sided').Ranked t-test. In this non-parametric test that compares the medians, all realizations are ranked together before being fed to a traditional t-test. Conover and Iman BID10 shows that the computed statistic is a monotonically increasing function of the statistic computed by the Wilcoxon MannWhitney test, making them really close. Implemented in our code. Bootstrap confidence interval test. In the bootstrap test, the sample is considered to be an approximation of the original distribution BID11. Given two observed samples (x 1, x 2) of size N, we obtain two bootstrap samples (x 1,x 2) of size N by sampling with replacement in (x 1, x 2) respectively and compute the difference in empirical means ∆x. This procedure is repeated a large number of times (e.g. 103). The distance between percentiles α×100 2 and 100(1− α 2) is considered to be the 100(1−α)% confidence interval around the true mean difference ∆µ. If it does not include 0, the test rejects the null hypothesis with confidence level α. This test does not require any assumptions on the performance distributions, but we will see it requires large sample sizes. Implementation: https://github.com/facebookincubator/bootstrapped. Permutation test. Under the null hypothesis, the realizations of both samples would come from distributions with the same mean. The empirical mean difference (∆x) should not be affected by the relabelling of the different realization (in average). The permutation test performs permutations of the realization labels and computes ∆x =x 1 −x 2. This procedure is repeated many times (e.g. 103). H 0 is rejected if the proportion of |∆x| that falls below the original difference |∆x| is higher than 1−α. Implemented in our code. This section compares the above statistical tests in terms of their false positive rates and statistical powers. A false positive rate estimates the probability to claim that two algorithms perform differently when H 0 holds. It impacts directly the reproducibility of a piece of research and should be as low as possible. Statistical power is the true positive rate and refers to the probability to find evidence for an existing effect. The following study is an extension of the one performed in BID12. We conduct experiments using models of RL distributions (analytic distributions) and true empirical RL distributions collected by running 192 trials of both SAC BID2 and TD3 BID3 on Half-Cheetah-v2 BID4 for 2M timesteps. Investigating the case of non-normal distributions. Several candidate distributions are selected to model RL performance distributions (FIG2): a standard normal distribution, a lognormal distribution and a bimodal distribution that is an even mixture of two normal distributions. All these distributions are tuned so that µ = 0, σ = 1. In addition we use two empirical distributions of size 192 collected from SAC and TD3.Investigating the case of unequal standard deviations. To investigate the effect of unequal standard deviations, we tune the distribution parameters to double the standard deviation of Algorithm 2 as compared to Algorithm 1. We also compare SAC and TD3 which have different standard deviations (σ T D3 = 1.15 σ SAC).Measuring false positive rates. To test for false positive rates α *, we simply enforce H 0 by aligning the central performances of the two distributions: µ 1 = µ 2 = 0 (the median for the MannWhitney test and the ranked t-test, the mean for others). Given one test, two distributions and a sample size, we sample x 1 and x 2 from distributions X 1, X 2 and compare them using the test with α = 0.05. We repeat this procedure N r = 10 3 times and estimate α * as the proportion of H 0 rejection. BID4 Using the spinning up implementation of OpenAI: https://github.com/openai/spinningupThe standard error of this estimate is: se(α *) = (α * (1−α *)/N r. It is smaller than the widths of the lines on the reported figures. This procedure is repeated for every test, every combination of distributions and for several sample sizes (see pseudo-code in the supplementary material).Measuring true positive rates (statistical power). Here, we enforce the alternative hypothesis H a by sampling x 1 from a given distribution centered in 0 (mean or median depending on the test), and x 2 from a distribution whose mean (resp. median) is shifted by an effect size ∆µ. Given one test, two distributions (the second being shifted) and the sample size, we repeat the procedure above and obtain an estimate of the true positive rate. Tables reporting the statistical powers for various effect sizes, sample sizes, tests and assumptions are made available in the supplementary . Same distributions, equal standard deviations., we can directly compare the mean estimates (the lines) to the significance level α = 0.05, the standard errors being smaller than the widths of these lines. BID5 α * is very large when using bootstrap tests, unless large sample sizes are used (>40). Using small sample sizes (<5), the permutation and the ranked t-test also show large α *. Results using two log-normal distributions show similar behaviors and can be found in the supplementary . Same distributions, unequal standard deviations. Here, we sample x 1 from a distribution, and x 2 from the same type of distribution with doubled standard deviation. Comparing two normal distributions with different standard deviation does not differ much from the case with equal standard deviations. Figure 4 (a) (bimodal distributions) shows that Mann-Whitney and ranked t-test (median tests) constantly overestimate α *, no matter the sample size (α * > 0.1). For log-normal distributions on the other hand (Figure 4(b) ), the false positive rate using these tests respects the confidence level (α * ≤ α) with sample sizes higher than N = 10. However, other tests tend to show large α *, even for large sample sizes (α * ≈ 0.07 up to N > 50).Different distributions, equal standard deviations. Now we compare samples coming from different distributions with equal standard deviations. Comparing normal and bimodal distributions of equal standard deviation does not impact much the false positive rates curves (similar to FIG3 (a)). However, FIG6 (a) and 5(b) show that when one of the two distributions is skewed (log-normal), the Mann-Whitney and the ranked t-test demonstrate very important false positive rate, a phenomenon that gets worse with larger sample sizes. Section 4.5 discusses why it might be the case. Different distributions, unequal standard deviations. We now combine different distributions and different standard deviations. As before, comparing a skewed distribution (log-normal) and a symmetric one leads to high false positive rates for the Mann-Whitney test and the ranked t-test BID5 We reproduced all the twice, hardly seeing any difference in the figures. All tests show similar estimations of statistical power. More than 50 samples are needed to detect a relative effect size = 0.5 with 80% probability, close to 20 with = 1 and a bit more than 10 with = 2. Tables reporting statistical power for Finally, we compare two empirical distributions obtained from running two RL algorithms (SAC, TD3) 192 times each, on Half-Cheetah. We observe a small increase in false positive rates when using the ranked t-test (Figure 7). The relative effect size estimated from the empirical distributions is = 0.80 (median), or = 0.93 (mean), in favor of SAC. For such relative effect sizes, the sample sizes required to achieve a statistical power of 0.8 are between 10 and 15 for tests comparing the mean and between 15 and 20 for tests comparing the median (see full table in supplementary ). Using a sample size N = 5 with the Welch's t-test, the effect size would need to be 3 to 4 times larger to be detected with 0.8 probability. No matter the distributions. From the above , it seems clear that the bootstrap test should never be used for sample sizes below N = 50 and the permutation test should never be used for sample sizes below N = 10. The bootstrap test in particular, uses the sample as an estimate of the true performance distribution. A small sample is a very noisy estimate, which leads to very high false positive rates. The ranked t-test shows a false positive rate of 0 and a statistical power of 0 when N = 2 in all conditions. As noted in BID12, comparing two samples of size N = 2 can in only four possible p-values (only 4 possible orders when ranked), none of which falls below α = 0.05. Such quantization issues make this test unreliable for small sample sizes, see BID12 for further comments and references on this issue. When distributions do not meet assumptions. In addition to the behaviors reported above, Section 4.2 shows that non-parametric tests (Mann-Whitney and ranked t-test) can demonstrate very high false positive rates when comparing a symmetric distribution with a skewed one (log-normal). This effect gets worse linearly with the sample size. When the sample size increases, the number of samples drawn in the skewed tail of the log-normal increases. All these realizations will be ranked above any realizations from the other distribution. Therefore, the larger the sample size, the more realization are ranked first in favor of the log-normal, which leads to a bias in the statistical test. This problem does not occur when two log-normal are compared to one another. Comparing a skewed distribution to a symmetric one violates the Mann-Whitney assumptions stating that distributions must have the same shape and spread. The false positive rates of Mann-Whitney and ranked t-test are also above the confidence level whenever a bimodal distribution is compared to another distribution. The traditional recommendation to use non-parametric tests when the distributions are not normal seems to be failing when the two distributions are different. Most robust tests. The t-test and the Welch's t-test were found to be more robust than others to violations of their assumptions. However, α * was found to be slightly above the required level (α * > α) when at least one of the two distributions is skewed (α * ≈ 0.1) no matter the sample size, and when one of the two distributions is bimodal, for small sample sizes N < 10. Welch's α * is always a bit lower than the t-test's α *.Statistical power. Except for the anomalies in small sample size mentioned above due to overconfident tests like the bootstrap or the permutation tests, statistical powers stay qualitatively stable no matter the distributions compared, or the test used: = 0.5: N ≈ 100; = 1: N ≈ 20 and = 2: N ≈ 5, 10. Measuring the performance of RL Algorithms. Before using any statistical test, one must obtain measures of performance. RL algorithms should ideally be evaluated offline. The algorithm performance after t steps is measured as the average of the returns over E evaluation episodes conducted independently from training, usually using a deterministic version of the current policy (e.g. E = 20). Evaluating agents by averaging performances over several training episodes in a much less interpretable performance measure and should be stated clearly. The collection of performance measures forms a learning curve. Representing learning curves. After obtaining a learning curve for each of the N runs, it can be rendered on a plot. At each evaluation, one can represent either the empirical mean or median. Whereas the empirical median directly represents the center of the collected sample, the empirical mean tries to model the sample as coming from a Gaussian distribution, and under this assumptions represents the maximum likelihood estimate of that center. Error bars should also be added to this plot. The standard deviation (SD) represents the variability of the performances, but is only representative when the values are approximately normally distributed. When it is not normal, one should prefer to represent interpercentile ranges (e.g. 10% − 90%). If the sample size is small (e.g. <10), the most informative solution is to represent all learning curves in addition to the mean or median. When performances are normally distributed, the standard error of the mean (SE) or confidence intervals can be used to represent estimates of the uncertainty on the mean estimate. Robust comparisons. Which test, which sample sizes? The in Section 4.2 advocate for the use of the Welch's t-test, which shows lower false positive rate and similar statistical powers than other tests. However, the false positive rate often remains superior to the confidence level α * > α when the distributions are not normal. When in doubt, we recommend using lower confidence levels α < 0.05 (e.g. α = 0.01) to ensure that α * < 0.05. The number of random seeds to be used to achieve a statistical power of 0.8 depends on the expected relative effect size: = 0.5: N ≈ 100; = 1: N ≈ 20 and = 2: N ≈ 5,10. The analysis of a real case comparing SAC and TD3 algorithms, required a sample size between N = 10 and N = 15 for a relatively strong effect = 0.93 when comparing the means, and about 5 more seeds when comparing the medians (= 0.80). Small sample sizes like N = 5 would require 3 to 4 times larger effects. A word on multiple comparisons. When performing multiple comparisons (e.g. between different pairs of algorithms evaluated in the same setting), the probability to have at least one false positive increases linearly with the number of comparisons n c. This probability is called the Family-Wise Error Rate (FWER). To correct for this effect, one must apply corrections. The Bonferroni correction for instance adapts the confidence level α Bonf. = α/n c BID13. This ensures that the FWER stays below the initial α. Using this corrections makes each test more conservative and decreases its statistical power. Comparing full learning curves. Instead of only comparing the final performances of the two algorithms after T timesteps in the environment, we can compare performances along learning. This consists in performing a statistical comparison for every evaluation step. This might reveal differences in speed of convergence and can provide more robust comparisons. Further discussions on how this relates to the problem of multiple comparison is given in the supplementary materials. In , this paper advocates for the use of Welch's t-test with low confidence level (α < 0.05) to ensure a false positive rate below α * < 0.05. The sample size must be selected carefully depending on the expected relative effect size. It also warns against the use of other unreliable tests, such as the bootstrap test (for N < 50), the Mann-Whitney and the ranked t-test (unless assumptions are carefully checked), or the permutation test (for N < 10). Using the t-test or the Welch's t-test with small sample sizes (<5) usually leads to high false positive rate and would require very large relative effect sizes (over = 2) to show good statistical power. Sample sizes above N = 20 generally meet the requirement of a 0.8 statistical power for a relative effect size = 1. Algorithm 1 represents the pseudo-code of the experiment. The whole code can be found at https: //github.com/ccolas/rl_stats. distributions refers to a list of pairs of distributions. When comparing tests for an equal distribution setting, the pairs represent twice the same type of distribution. When comparing for an unequal variance setting, the standard deviation of the second distribution is doubled. The number of repetitions is set to 10.000. The rejection variable refers to the rejection of the null hypothesis. The false positive error rates can be found in _array [for i_t, test in tests do 5: for i_e, effect_size in effect_sizes do 6: for i_ss, N in sample_sizes do for i_r = 1: nb_repets do 9: distrib BID0 .shift(effect) 10: sample1 = distrib.sample(N) 11: sample2 = distrib BID0.sample(N) 12: rejection_list.append(test.test(sample1, sample2, α)) 13: _array[i_d, i_t, i_e, i_ss] = mean(rejection_list) The correction to apply when comparing two learning curves depends 1) on the number of comparisons, 2) on the criteria that is used to conclude whether an algorithm is better than the other. The criteria used to draw a must be decided before running any test. An example can be: if when comparing the last 100 performance measures of the two algorithms, more than 50 comparisons show a significant difference, then Algorithm 1 is better than Algorithm 2. In that case, the number of comparisons performed is N c = 100, and the criterion is N rejection > N crit = 50. We want to constrain the probability FWER that our criterion is met by pure chance to a confidence level α=0.05. This probability is: FWER = α×N c /N crit. To make it satisfy FWER = 0.05, we need to correct α such as α corrected = α×N crit /N c (α corrected = α/2 in our case). Table 6: Statistical power when comparing samples from two bimodal distribution with different standard deviation. The first is centered in 0 (µ 1 = 0, σ 1 = 1 mean or median depending on the test), the other shifted by the relative effect size (µ 2 = σ pool, σ 2 = 2). Both have same standard deviation σ 1 = σ 2 = 1. Each represents the percentage of true positive over 10.000 repetitions. In bold are satisfying a true positive rate above 0.8.

This paper compares statistical tests for RL comparisons (false positive, statistical power), checks robustness to assumptions using simulated distributions and empirical distributions (SAC, TD3), provides guidelines for RL students and researchers.

represent the short summary for finding the corresponding scientific article (long)

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.