Question: Represent the article for finding a single-sentence TLDR:

Question: In partially observable (PO) environments, deep reinforcement learning (RL) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how to improve the policy. In this study, we propose an RL algorithm for solving PO tasks. Our method comprises two parts: a variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM. The proposed algorithm was tested in two types of PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization. Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner. Model-free deep reinforcement learning (RL) algorithms have been developed to solve difficult control and decision-making tasks by self-exploration (; ;). While various kinds of fully observable environments have been well investigated, recently, partially observable (PO) environments (; ; ;) have commanded greater attention, since real-world applications often need to tackle incomplete information and a non-trivial solution is highly desirable. There are many types of PO tasks; however, those that can be solved by taking the history of observations into account are more common. These tasks are often encountered in real life, such as videos games that require memorization of previous events and robotic control using real-time images as input . While humans are good at solving these tasks by extracting crucial information from the past observations, deep RL agents often have difficulty acquiring satisfactory policy and achieving good data efficiency, compared to those in fully observable tasks . For solving such PO tasks, several categories of methods have been proposed. One simple, straightforward solution is to include a history of raw observations in the current "observation" . Unfortunately, this method can be impractical when decision-making requires a long-term memory because dimension of observation become unacceptably large if a long history is included. Another category is based on model-free RL methods with recurrent neural networks (RNN) as function approximators (; 1991; ; ;), which is usually more tractable to implement. In this case, RNNs need to tackle two problems simultaneously : learning representation (encoded by hidden states of the RNN) of the underlying states of the environment from the state-transition data, and learning to maximize returns using the learned representation. As most RL algorithms use a bootstrapping strategy to learn the expected return and to improve the policy , it is challenging to train the RNN stably and efficiently, since RNNs are relatively more difficult to train than feedforward neural networks. The third category considers learning a model of the environment and estimating a belief state, extracted from a sequence of state-transitions (; ;). The belief state is an agent-estimated variable encoding underlying states of the environment that determines state-transitions and rewards. Perfectly-estimated belief states can thus be taken as "observations" of an RL agent that contains complete information for solving the task. Therefore, solving a PO task is segregated into a representation learning problem and a fully observable RL problem. Since fully observable RL problems have been well explored by the RL community, the critical challenge here is how to estimate the belief state. In this study, we developed a variational recurrent model (VRM) that models sequential observations and rewards using a latent stochastic variable. The VRM is an extension of the variational recurrent neural network (VRNN) model that takes actions into account. Our approach falls into the third category by taking the internal states of the VRM together with raw observations as the belief state. We then propose an algorithm to solve PO tasks by training the VRM and a feed-forward RL controller network, respectively. The algorithm can be applied in an end-to-end manner, without fine tuning of a hyperparameters. We then experimentally evaluated the proposed algorithm in various PO versions of robotic control tasks. The agents showed substantial policy improvement in all tasks, and in some tasks the algorithm performed essentially as in fully observable cases. In particular, our algorithm demonstrates greater performance compared to alternative approaches in environments where only velocity information is observable or in which long-term memorization is needed. Typical model-based RL approaches utilize learned models for dreaming, i.e. generating statetransition data for training the agent (; ;) or for planning of future state-transitions (; ;). This usually requires a well-designed and finely tuned model so that its predictions are accurate and robust. In our case, we do not use VRMs for dreaming and planning, but for auto-encoding state-transitions. Actually, PO tasks can be solved without requiring VRMs to predict accurately (see Appendix E). This distinguishes our algorithm from typical model-based RL methods. The work our method most closely resembles is known as stochastic latent actor-critic , in which a latent variable model is trained and uses the latent state as the belief state for the critic. SLAC showed promising using pixels-based robotic control tasks, in which velocity information needs to be inferred from third-person images of the robot. Here we consider more general PO environments in which the reward may depend on a long history of inputs, e.g., in a snooker game one has to remember which ball was potted previously. The actor network of SLAC did not take advantage of the latent variable, but instead used some steps of raw observations as input, which creates problems in achieving long-term memorization of reward-related state-transitions. Furthermore, SLAC did not include raw observations in the input of the critic, which may complicate training the critic before the model converges. The scope of problems we study can be formulated into a framework known as partially observable Markov decision processes (POMDP) . POMDPs are used to describe decision or control problems in which a part of underlying states of the environment, which determine state-transitions and rewards, cannot be directly observed by an agent. A POMDP is usually defined as a 7-tuple (S, A, T, R, X, O, γ), in which S is a set of states, A is a set of actions, and T: S × A → p(S) is the state-transition probability function that determines the distribution of the next state given current state and action. The reward function R: S × A → R decides the reward during a state-transition, which can also be probabilistic. Moreover, X is a set of observations, and observations are determined by the observation probability function O: S × A → p(X). By defining a POMDP, the goal is to maximize expected discounted future rewards t γ t r t by learning a good strategy to select actions (policy function). Our algorithm was designed for general POMDP problems by learning the representation of underlying states s t ∈ S via modeling observation-transitions and reward functions. However, it is expected to work in PO tasks in which s t or p(s t) can be (at least partially) estimated from the history of observations x 1:t. To model general state-transitions that can be stochastic and complicated, we employ a modified version of the VRNN . The VRNN was developed as a recurrent version of the variational auto-encoder , composed of a variational generation model and a variational inference model. It is a recurrent latent variable model that can learn to encode and predict complicated sequential observations x t with a stochastic latent variable z t. The generation model predicts future observations given the its internal states, where f s are parameterized mappings, such as feed-forward neural networks, and d t is the state variable of the RNN, which is recurrently updated by The inference model approximates the latent variable z t given x t and d t. For sequential data that contain T time steps, learning is conducted by maximizing the evidence lower bound ELBO, like that in a VEA , where where p and q are parameterized PDFs of z t by the generative model and the inference model, respectively. In a POMDP, a VRNN can be used to model the environment and to represent underlying states in its state variable d t. Thus an RL agent can benefit from a well-learned VRNN model since d t provides additional information about the environment beyond the current raw observation x t. Soft actor-critic (SAC) is a state-of-the-art model-free RL that uses experience replay for dynamic programming, which been tested on various robotic control tasks and that shows promising performance (a; b). A SAC agent learns to maximize reinforcement returns as well as entropy of its policy, so as to obtain more rewards while keeping actions sufficiently stochastic. A typical SAC implementation can be described as follows. The state value function V (s), the state-action value function Q(s, a) and the policy function π(a|s) are parameterized by neural networks, indicated by ψ, λ, η, respectively. Also, an entropy coefficient factor (also known as the temperature parameter), denoted by α, is learned to control the degree of stochasticity of the policy. The parameters are learned by simultaneously minimizing the following loss functions. where B is the replay buffer from which s t is sampled, and H tar is the target entropy. To compute the gradient of J π (η) (Equation. 7), the reparameterization trick is used on action, indicated by a η (s t). Reparameterization of action is not required in minimizing J(α) (Equation. 8) since log π η (a|s t) does not depends on α. SAC was originally developed for fully observable environments; thus, the raw observation at the current step x t was used as network input. In this work, we apply SAC in PO tasks by including the state variable d t of the VRNN in the input of function approximators of both the actor and the critic. An overall diagram of the proposed algorithm is summarized in Fig. 1(a), while a more detailed computational graph is plotted in Fig. 2. We extend the original VRNN model to the proposed VRM model by adding action feedback, i.e., actions taken by the agent are used in the inference model and the generative model. Also, since we are modeling state-transition and reward functions, we include the reward r t−1 in the current raw observation x t for convenience. Thus, we have the inference model (Fig. 1(c) ), denoted by φ, as The generative model (Fig. 1(b) ), denoted by θ here, is For building recurrent connections, the choice of RNN types is not limited. In our study, the longshort term memory (LSTM) is used since it works well in general cases. So we have As in training a VRNN, the VRM is trained by maximizing an evidence lower bound (Fig. 1(c) ) In practice, the first term E q φ [log p θ (x t |z 1:t, x 1:t−1)] can be obtained by unrolling the RNN using the inference model (Fig. 1(c) ) with sampled sequences of x t. Since q φ and p θ are parameterized Gaussian distributions, the KL-divergence term can be analytically expressed as For computation efficiency in experience replay, we train a VRM by sampling minibatchs of truncated sequences of fixed length, instead of whole episodes. Details are found in Appendix A.1. Since training of a VRM is segregated from training of the RL controllers, there are several strategies for conducting them in parallel. For the RL controller, we adopted a smooth update strategy as in Haarnoja et al. (2018a), i.e., performing one time of experience replay every n steps. To train the VRM, one can also conduct smooth update. However, in that case, RL suffers from instability of the representation of underlying states in the VRM before it converges. Also, stochasticity of RNN state variables d can be meaninglessly high at early stage of training, which may create problems in RL. Another strategy is to pre-train the VRM for abundant epochs only before RL starts, which unfortunately, can fail if novel observations from the environment appear after some degree of policy improvement. Moreover, if pre-training and smooth update are both applied to the VRM, RL may suffer from a large representation shift of the belief state. To resolve this conflict, we propose using two VRMs, which we call the first-impression model and the keep-learning model, respectively. As the names suggest, we pre-train the first-impression model and stop updating it when RL controllers and the keep-learning model start smooth updates. Then we take state variables from both VRMs, together with raw observations, as input for the RL controller. We found that this method yields better overall performance than using a single VRM (Appendix C). Initialize the first-impression VRM M f and the keep-learning VRM M k, the RL controller C, and the replay buffer D, global step t ← 0. repeat Initialize an episode, assign M with zero initial states. while episode not terminated do Sample an action a t from π(a t |d t, x t) and execute a t, t ← t + 1. Compute 1-step forward of both VRMs using inference models. if t == step start RL then For N epochs, sample a minibatch of samples from B to update M f (Eq. 11). end if if t > step start RL and mod(t, train interval KLV RM) == 0 then Sample a minibatch of samples from B to update M k (Eq. 5, 6, 7, 8). end if if t > step start RL and mod(t, train interval RL) == 0 then Sample a minibatch of samples from B to update R (Eq. 11). end if end while until training stopped As shown in Fig. 1(a), we use multi-layer perceptrons (MLP) as function approximators for V, Q, respectively. Inputs for the Q t network are (x t, d t, a t), and V t is mapped from (x t, d t). Following Haarnoja et al. (2018a), we use two Q networks λ 1 and λ 2 and compute Q = min(Q λ1, Q λ2) in Eq. 5 and 7 for better performance and stability. Furthermore, we also used a target value network for computing V in Eq. 6 as in Haarnoja et al. (2018a). The policy function π η follows a parameterized where µ η and σ η are also MLPs. In the execution phase (Fig. 1(b) ), observation and reward x t = (X t, r t−1) are received as VRM inputs to compute internal states d t using inference models. Then, the agent selects an action, sampled from π η (a t |d t, x t), to interact with the environment. To train RL networks, we first sample sequences of steps from the replay buffer as minibatches; thus, d t can be computed by the inference models using recorded observationsx t and actionsā t (See Appendix A.1.2). Then RL networks are updated by minimizing the loss functions with gradient descent. Gradients stop at d t so that training of RL networks does not involve updating VRMs. To empirically evaluate our algorithm, we performed experiments in a range of (partially observable) continuous control tasks and compared it to the following alternative algorithms. The overall procedure is summarized in Algorithm 1. For the RL controllers, we adopted hyperparameters from the original SAC implementation (b). Both the keep-learning and first-impression VRMs were trained using learning rate 0.0008. We pre-trained the first-impression VRM for 5,000 epochs, and updated the keep-learning VRM every 5 steps. Batches of size 4, each containing a sequence of 64 steps, were used for training both the VRMs and the RL controllers. All tasks used the same hyperparameters (Appendix A.1). • SAC-MLP: The vanilla soft actor-critic implementation (a; b), in which each function is approximated by a 2-layer MLP taking raw observations as input. • SAC-LSTM: Soft actor-critic with recurrent networks as function approximators, where raw observations are processed through an LSTM layer followed by 2 layers of MLPs. This allows the agent to make decisions based on the whole history of raw observations. In this case, the network has to conduct representation learning and dynamic programming collectively. Our algorithm is compared with SAC-LSTM to demonstrate the effect of separating representation learning from dynamic programming. Note that in our algorithm, we apply pre-training of the first-impression model. For a fair comparison, we also perform pre-training for the alternative algorithm with the same epochs. For SAC-MLP and SAC-LSTM, pre-training is conducted on RL networks; while for SLAC, its model is pre-trained. The Pendlum and CartPole tasks are the classic control tasks for evaluating RL algorithms (Fig. 3, Left). The CartPole task requires learning of a policy that prevents the pole from falling down and keeps the cart from running away by applying a (1-dimensional) force to the cart, in which observable information is the coordinate of the cart, the angle of the pole, and their derivatives w.r.t time (i.e., velocities). For the Pendulum task, the agent needs to learn a policy to swing an inverse-pendulum up and to maintain it at the highest position in order to obtain more rewards. We are interested in classic control tasks because they are relatively easy to solve when fully observable, and thus the PO cases can highlight the representation learning problem. Experiments were performed in these two tasks, as well as their PO versions, in which either velocities cannot be observed or only velocities can be observed. The latter case is meaningful in real-life applications because an agent may not be able to perceive its own position, but can estimate its speed. As expected, SAC-MLP failed to solve the PO tasks (Fig. 3). While our algorithm succeeded in learning to solve all these tasks, SAC-LSTM showed poorer performance in some of them. In particular, in the pendulum task with only angular velocity observable, SAC-LSTM may suffer from the periodicity of the angle. SLAC performed well in the CartPole tasks, but showed less satisfactory sample efficiency in the Pendulum tasks. To examine performance of the proposed algorithm in more challenging control tasks with higher degrees of freedom (DOF), we also evaluated performance of the proposed algorithm in the OpenAI Roboschool environments . The Roboschool environments include a number of continuous robotic control tasks, such as teaching a multiple-joint robot to walk as fast as possible without falling down (Fig. 4, Left). The original Roboschool environments are nearly fully observable since observations include the robot's coordinates and (trigonometric functions of) joint angles, as well as (angular and coordinate) velocities. As in the PO classic control tasks, we also performed experiments in the PO versions of the Roboschool environments. Using our algorithm, experimental (Fig. 4) demonstrated substantial policy improvement in all PO tasks (visualization of the trained agents is in Appendix D). In some PO cases, the agents achieved comparable performance to that in fully observable cases. For tasks with unobserved velocities, our algorithm performed similarly to SAC-LSTM. This is because velocities can be simply estimated by one-step differences in robot coordinates and joint angles, which eases representation learning. However, in environments where only velocities can be observed, our algorithm significantly outperformed SAC-LSTM, presumably because SAC-LSTM is less efficient at encoding underlying states from velocity observations. Also, we found that learning of a SLAC agent was unstable, i.e., it sometimes could acquire a near-optimal policy, but often its policy converged to a poor one. Thus, average performance of SLAC was less promising than ours in most of the PO robotic control tasks. Another common type of PO task requires long-term memorization of past events. To solve these tasks, an agent needs to learn to extract and to remember critical information from the whole history of raw observations. Therefore, we also examined our algorithm and other alternatives in a long-term memorization task known as the sequential target reaching task , in which a robot agent needs to reach 3 different targets in a certain sequence (Fig. 5, Left). The robot can control its two wheels to move or turn, and will get one-step small, medium, and large rewards when it reaches the first, second, and third targets, respectively, in the correct sequence. The robot senses distances and angles from the 3 targets, but does not receive any signal indicating which target to reach. In each episode, the robot's initial position and those of the three targets are randomly initialized. In order to obtain rewards, the agent needs to infer the current correct target using historical observations. We found that agents using our algorithm achieved almost 100% success rate (reaching 3 targets in the correct sequence within maximum steps). SAC-LSTM also achieved similar success rate after convergence, but spent more training steps learning to encode underlying goal-related information from sequential observations. Also, SLAC struggled hard to solve this task since its actor only received a limited steps of observations, making it difficult to infer the correct target. One of the most concerned problems of our algorithm is that input of the RL controllers can experience representation change, because the keep-learning model is not guaranteed to converge if novel observation appears due to improved policy (e.g. for a hopper robot, "in-the-air" state can only happen after it learns to hop). To empirically investigate how convergence of the keep-learning VRM affect policy improvement, we plot the loss functions (negative ELBOs) of the the keep-learning VRM for 3 example tasks (Fig. 6). For a simpler task (CartPole), the policy was already near optimal before the VRM fully converged. We also saw that the policy was gradually improved after the VRM mostly converged (RoboschoolAnt -no velocities), and that the policy and the VRM were being improved in parallel (RoboschoolAnt -velocities only). The suggested that policy could be improved with sufficient sample efficiency even the keep-learning VRM did not converge. This can be explained by that the RL controller also extract information from the first-impression model and the raw observations, which did not experience representation change during RL. Indeed, our ablation study showed performance degradation in many tasks without the first-impression VRM (Appendix C). In this paper, we proposed a variational recurrent model for learning to represent underlying states of PO environments and the corresponding algorithm for solving POMDPs. Our experimental demonstrate effectiveness of the proposed algorithm in tasks in which underlying states cannot be simply inferred using a short sequence of observations. Our work can be considered an attempt to understand how RL benefits from stochastic Bayesian inference of state-transitions, which actually happens in the brain , but has been considered less often in RL studies. We used stochastic models in this work which we actually found perform better than deterministic ones, even through the environments we used are deterministic (Appendix C). The VRNN can be replaced with other alternatives to potentially improve performance, although developing model architecture is beyond the scope of the current study. Moreover, a recent study showed a novel way of inference using back-propagation of prediction errors, which may also benefit our future studies. Many researchers think that there are two distinct systems for model-based and model-free RL in the brain (Gläscher et al., 2010;) and a number of studies investigated how and when the brain switches between them . suggested that the hippocampus can learn a successor representation of the environment that benefits both model-free and model-based RL, contrary to the aforementioned conventional view. We further propose another possibility, that a model is learned, but not used for planning or dreaming. This blurs the distinction between model-based and model-free RL. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1856-1865, 2018a. In this section we describe the details of implementing our algorithm as well as the alternative ones. Summaries of hyperparameters can be found in Table 1 and 2. The first-impression model and the keep-learning model adopted the same architecture. Size of d and z is 256 and 64, respectively. We used one-hidden-layer fully-connected networks with 128 hidden neurons for the inference models µ φ,t, σ 2 φ,t = φ(x t, d t−1, a t−1), as well as for µ θ,t, σ 2 θ,t = θ prior (d t−1, a t−1) in the generative models. For the decoder µ x,t, σ 2 x,t = θ decoder (z t, d t−1) in the generative models, we used 2-layers MLPs with 128 neurons in each layer. The input processing layer f x is also an one-layer MLP with size-128. For all the Gaussian variables, output functions for mean are linear and output functions for variance are softplus. Other activation functions of the VRMs are tanh. The RL controllers are the same as those in SAC-MLP (Section A.2.1) except that network inputs are raw observations together with the RNN states from the first-impression model and the keep-learning model. To train the VRMs, one can use a number of entire episodes as a mini-batch, using zero initial states, as in. However, when tackling with long episodes (e.g. there can be 1,000 steps in each episode in the robotic control tasks we used) or even infinite-horizon problems, the computation consumption will be huge in back-propagation through time (BPTT). For better computation efficiency, we used 4 length-64 sequences for training the RNNs, and applied the burn-in method for providing the initial states , or more specifically, unrolling the RNNs using a portion of the replay sequence (burn-in period, up to 64 steps in our case) from zero initial states. We assume that proper initial states can be obtained in this way. This is crucial for the tasks that require long-term memorization, and is helpful to reduce bias introduces by incorrect initial states in general cases. A.2.1 SAC-MLP We followed the original implementation of SAC in (a) including hyperparameters. However, we also applied automatic learning of the entropy coefficient α (inverse of the the reward scale in Haarnoja et al. (2018a) ) as introduced by the authors in Haarnoja et al. (2018b) to avoid tuning the reward scale for each task. To apply recurrency to SAC's function approximators, we added an LSTM network with size-256 receiving raw observations as input. The function approximators of actor and critic were the same as those in SAC except receiving the LSTM's output as input. The gradients can pass through the LSTM so that the training of the LSTM and MLPs were synchronized. The training the network also followed Section A.1.2. We mostly followed the implementation of SLAC explained in the authors' paper . One modification is that since their work was using pixels as observations, convolutional neural networks (CNN) and transposed CNNs were chosen for input feature extracting and output decoding layers; in our case, we replaced the CNN and transposed CNNs by 2-layers MLPs with 256 units in each layer. In addition, the authors set the output variance σ 2 y,t for each image pixel as 0.1. However, σ 2 y,t = 0.1 can be too large for joint states/velocities as observations. We found that it will lead to better performance by setting σ y,t as trainable parameters (as that in our algorithm). We also used a 2-layer MLP with 256 units for approximating σ y (x t, d t−1). To avoid network weights being divergent, all the activation functions of the model were tanh except those for outputs. For the robotic control tasks and the Pendulum task, we used environments (and modified them for PO versions) from OpenAI Gym . The CartPole environment with a continuous action space was from , and the codes for the sequential target reaching tasks were provided by the authors . In the no-velocities cases, velocity information was removed from raw observations; while in the velocities-only cases, only velocity information was retained in raw observations. We summarize key information of each environment in Table 3. The performance curves were obtained in evaluation phases in which agents used same policy but did not update networks or record state-transition data. Each experiment was repeated using 5 different random seeds. This section demonstrated a ablation study in which we compared the performance of the proposed algorithm to the same but with some modification: • With a single VRM. In this case, we used only one VRM and applied both pre-training and smooth update to it. • Only first-impression model. In this case, only the first-impression model was used and pre-trained. • Only keep-learning model. In this case, only the keep-learning model was used and smooth-update was applied. • Deterministic model. In this case, the first-imporession model and the keep-learning model were deterministic RNNs which learned to model the state-transitions by minimizing mean-square error between prediction and observations instead of ELBO. The network architecture was mostly the same as the VRM expect that the inference model and the generative model were merged into a deterministic one. The learning curves are shown in Fig. 7. It can be seen that the proposed algorithm consistently performed similar as or better than the modified ones. Here we show actual movements of the trained robots in the PO robotic control tasks (Fig. 8). It can be seen that the robots succeeded in learning to hop or walk, although their policy may be sub-optimal. As we discussed in Section 2, our algorithm relies mostly on encoding capacity of models, but does not require models to make accurate prediction of future observations. Fig. 9 shows open-loop (using the inference model to compute the latent variable z) and close-loop (purely using the generative model) prediction of raw observation by the keep-learning models of randomly selected trained agents. Here we showcase "RoboschoolHopper -velocities only" and "Pendulum -no velocities" because in these tasks our algorithm achieved similar performance to those in fully-observable versions (Fig. 4), although the prediction accuracy of the models was imperfect. To empirically show how choice of hyperparameters of the VRMs affect RL performance, we conducted experiments using hyperparameters different from those used in the main study. More specifically, the learning rate for both VRMs was randomly selected from {0.0004, 0.0006, 0.0008, 0.001} and the sequence length was randomly selected from {16, 32, 64} (the batch size was 256/(sequence length) to ensure that the total number of samples in a batch was 256 which matched with the alternative approaches). The other hyperparameters were unchanged. The can be checked in Fig 10 for all the environments we used. The overall performance did not significantly change using different, random hyperparameters of the VRMs, although we could observe significant performance improvement (e.g. RoboshoolWalker2d) or degradation (e.g. RoboshoolHopper -velocities only) in a few tasks using different haperparameters. Therefore, the representation learning part (VRMs) of our algorithm does not suffer from high sensitivity to hyperparameters. This can be explained by the fact that we do not use a bootstrapping (e.g. the estimation of targets of value functions depends on the estimation of value functions) update rule to train the VRMs. G SCALABILITY Table 4 showed scalability of our algorithm and the alternative ones. Table 4: Wall-clock time and number of parameters of our algorithm and the alternative ones. The working environment was a desktop computer using Intel i7-6850K CPU and the task is "Velocitiesonly RoboschoolHopper". The wall-clock time include training the first-impression VRM or pretrainings.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A deep RL algorithm for solving POMDPs by auto-encoding the underlying states using a variational recurrent model

represent the short summary for finding the corresponding scientific article (long).

A structured latent-variable approach that adds discrete control states within a standard autoregressive neural paradigm to provide arbitrary grounding of internal model decisions, without sacrificing any representational power of neural models.


Question: Represent the article for finding a single-sentence TLDR

Question: This paper formalises the problem of online algorithm selection in the context of Reinforcement Learning (RL). The setup is as follows: given an episodic task and a finite number of off-policy RL algorithms, a meta-algorithm has to decide which RL algorithm is in control during the next episode so as to maximize the expected return. The article presents a novel meta-algorithm, called Epochal Stochastic Bandit Algorithm Selection (ESBAS). Its principle is to freeze the policy updates at each epoch, and to leave a rebooted stochastic bandit in charge of the algorithm selection. Under some assumptions, a thorough theoretical analysis demonstrates its near-optimality considering the structural sampling budget limitations. ESBAS is first empirically evaluated on a dialogue task where it is shown to outperform each individual algorithm in most configurations. ESBAS is then adapted to a true online setting where algorithms update their policies after each transition, which we call SSBAS. SSBAS is evaluated on a fruit collection task where it is shown to adapt the stepsize parameter more efficiently than the classical hyperbolic decay, and on an Atari game, where it improves the performance by a wide margin. Reinforcement Learning (RL, BID18) is a machine learning framework for optimising the behaviour of an agent interacting with an unknown environment. For the most practical problems, such as dialogue or robotics, trajectory collection is costly and sample efficiency is the main key performance indicator. Consequently, when applying RL to a new problem, one must carefully choose in advance a model, a representation, an optimisation technique and their parameters. Facing the complexity of choice, RL and domain expertise is not sufficient. Confronted to the cost of data, the popular trial and error approach shows its limits. We develop an online learning version (; BID1 of Algorithm Selection (AS, BID15 ; BID17 BID5). It consists in testing several algorithms on the task and in selecting the best one at a given time. For clarity, throughout the whole article, the algorithm selector is called a meta-algorithm, and the set of algorithms available to the meta-algorithm is called a portfolio. The meta-algorithm maximises an objective function such as the RL return. Beyond the sample efficiency objective, the online AS approach besides addresses four practical problems for online RL-based systems. First, it improves robustness: if an algorithm fails to terminate, or outputs to an aberrant policy, it will be dismissed and others will be selected instead. Second, convergence guarantees and empirical efficiency may be united by covering the empirically efficient algorithms with slower algorithms that have convergence guarantees. Third, it enables curriculum learning: shallow models control the policy in the early stages, while deep models discover the best solution in late stages. And four, it allows to define an objective function that is not an RL return. A fair algorithm selection implies a fair budget allocation between the algorithms, so that they can be equitably evaluated and compared. In order to comply with this requirement, the reinforcement algorithms in the portfolio are assumed to be off-policy, and are trained on every trajectory, regardless which algorithm controls it. Section 2 provides a unifying view of RL algorithms, that allows information sharing between algorithms, whatever their state representations and optimisation techniques. It also formalises the problem of online selection of off-policy RL algorithms. Next, Section 3 presents the Epochal Stochastic Bandit AS (ESBAS), a novel meta-algorithm addressing the online off-policy RL AS problem. Its principle relies on a doubling trick: it divides the time-scale into epochs of exponential length inside which the algorithms are not allowed to update their policies. During each epoch, the algorithms have therefore a constant policy and a stochastic multi-armed bandit can be in charge of the AS with strong pseudo-regret theoretical guaranties. A thorough theoretical analysis provides for ESBAS upper bounds. Then, Section 4 evaluates ESBAS on a dialogue task where it outperforms each individual algorithm in most configurations. Afterwards, in Section 5, ESBAS, which is initially designed for a growing batch RL setting, is adapted to a true online setting where algorithms update their policies after each transition, which we call SSBAS. It is evaluated on a fruit collection task where it is shown to adapt the stepsize parameter more efficiently than the classical hyperbolic decay, and on Q*bert, where running several DQN with different network size and depth in parallel allows to improve the final performance by a wide margin. Finally, Section 6 concludes the paper with prospective ideas of improvement. Stochastic environment a(t) o(t + 1) r(t + 1)Figure 1: RL framework: after performing action a(t), the agent perceives observation o(t + 1) and receives reward r(t + 1).The goal of this section is to enable information sharing between algorithms, even though they are considered as black boxes. We propose to share their trajectories expressed in a universal format: the interaction process. Reinforcement Learning (RL) consists in learning through trial and error to control an agent behaviour in a stochastic environment: at each time step t ∈ N, the agent performs an action a(t) ∈ A, and then perceives from its environment a signal o(t) ∈ Ω called observation, and receives a reward r(t) ∈ R, bounded between R min and R max. Figure 1 illustrates the RL framework. This interaction process is not Markovian: the agent may have an internal memory. In this article, the RL problem is assumed to be episodic. Let us introduce two time scales with different notations. First, let us define meta-time as the time scale for AS: at one meta-time τ corresponds a meta-algorithm decision, i.e. the choice of an algorithm and the generation of a full episode controlled with the policy determined by the chosen algorithm. Its realisation is called a trajectory. Second, RL-time is defined as the time scale inside a trajectory, at one RL-time t corresponds one triplet composed of an observation, an action, and a reward. Let E denote the space of trajectories. A trajectory ε τ ∈ E collected at meta-time τ is formalised as a sequence of (observation, action, reward) triplets: ε τ = o τ (t), a τ (t), r τ (t) t∈ 1,|ετ | ∈ E, where |ε τ | is the length of trajectory ε τ. The objective is, given a discount factor 0 ≤ γ < 1, to generate trajectories with high discounted cumulative reward, also called return, and noted µ(ε τ) = |ετ | t=1 γ t−1 r τ (t). Since γ < 1 and R is bounded, the return is also bounded. The trajectory set at meta-time T is denoted by D T = {ε τ} τ ∈ 1,T ∈ E T. A sub-trajectory of ε τ until RL-time t is called the history at RL-time t and written ε τ (t) with t ≤ |ε τ |. The history records what happened in episode ε τ until RL-time t: ε τ (t) = o τ (t), a τ (t), r τ (t) t ∈ 1,t ∈ E.The goal of each RL algorithm α is to find a policy π *: E → A which yields optimal expected returns. Such an algorithm α is viewed as a black box that takes as an input a trajectory set D ∈ E +, where E + is the ensemble of trajectory sets of undetermined size: DISPLAYFORM0, and that outputs a policy π α D. Consequently, a RL algorithm is formalised as follows: α: DISPLAYFORM1 Such a high level definition of the RL algorithms allows to share trajectories between algorithms: a trajectory as a sequence of observations, actions, and rewards can be interpreted by any algorithm in its own decision process and state representation. For instance, RL algorithms classically rely on an MDP defined on a explicit or implicit state space representation DISPLAYFORM2 on the trajectories projected on its state space representation. Off-policy RL optimisation techniques compatible with this approach are numerous in the literature BID20; BID10. As well, any post-treatment of the state set, any alternative decision process BID9, and any off-policy algorithm may be used. The algorithms are defined here as black boxes and the considered meta-algorithms will be indifferent to how the algorithms compute their policies, granted they satisfy the off-policy assumption. Pseudo-code 1: Online RL AS setting DISPLAYFORM0 Generate trajectory ε τ with policy π DISPLAYFORM1 The online learning approach is tackled in this article: different algorithms are experienced and evaluated during the data collection. Since it boils down to a classical exploration/exploitation trade-off, multi-armed bandit have been used for combinatorial search AS (; BID1 and evolutionary algorithm meta-learning . The online AS problem for off-policy RL is novel and we define it as follows: DISPLAYFORM2 is the current trajectory set;• P = {α k} k∈ 1,K is the portfolio of off-policy RL algorithms;• µ: E → R is the objective function, generally set as the RL return. Pseudo-code 1 formalises the online RL AS setting. A meta-algorithm is defined as a function from a trajectory set to the selection of an algorithm: σ: E + → P. The meta-algorithm is queried at each meta-time τ = |D τ −1 |+1, with input D τ −1, and it ouputs algorithm σ (D τ −1) = σ(τ) ∈ P controlling with its policy π σ(τ) Dτ−1 the generation of the trajectory ε τ in the stochastic environment. Figure 2 illustrates the algorithm with a diagram flow. The final goal is to optimise the cumulative expected return. It is the expectation of the sum of rewards obtained after a run of T trajectories: DISPLAYFORM3 DISPLAYFORM4 Return µ(ε τ): DISPLAYFORM5 Figure 2: Algorithm selection for reinforcement learning flow diagram expectations. The outside expectation E σ assumes the meta-algorithm σ fixed and averages over the trajectory set generation and the corresponding algorithms policies. The inside expectation Eµ assumes the policy fixed and averages over its possible trajectories in the stochastic environment. Nota bene: there are three levels of decision: meta-algorithm σ selects algorithm α that computes policy π that is in control. In this paper, the focus is at the meta-algorithm level. In this paper, we focus on sample efficiency, where a sample is meant to be a trajectory. This is motivated by the following reasons. First, in most real-world systems, the major regret is on the task failure. The time expenditure is only a secondary concern that is already assessed by the discount factor dependency in the return. Second, it would be inconsistent to consider regret on a different time scale as the algorithm selection. Also, policy selection on non-episodic RL is known as a very difficult task where state-of-the-art algorithms only obtain regrets of the order of O(√ T log(T)) on stationary policies. Third, the regret on the decision steps cannot be assessed, since the rewards are discounted in the RL objective function. And finally, the bandit rewards (defined as the objective function in Section 2.2) may account for the length of the episode. In order to evaluate the meta-algorithms, let us formulate two additional notations. First, the optimal expected return Eµ * ∞ is defined as the highest expected return achievable by a policy of an algorithm in portfolio P. Second, for every algorithm α in the portfolio, let us define σ α as its canonical metaalgorithm, i.e. the meta-algorithm that always selects algorithm α: ∀τ, σ α (τ) = α. The absolute pseudo-regret ρ σ abs (T) defines the regret as the loss for not having controlled the trajectory with an optimal policy: DISPLAYFORM0 It is worth noting that an optimal meta-algorithm will unlikely yield a null regret because a large part of the absolute pseudo-regret is caused by the sub-optimality of the algorithm policies when the trajectory set is still of limited size. Indeed, the absolute pseudo-regret considers the regret for not selecting an optimal policy: it takes into account both the pseudo-regret of not selecting the best algorithm and the pseudo-regret of the algorithms for not finding an optimal policy. Since the metaalgorithm does not interfere with the training of policies, it ought not account for the pseudo-regret related to the latter. Related to AS for RL, BID16 use meta-learning to tune a fixed RL algorithm in order to fit observed animal behaviour, which is a very different problem to ours. In Cauwet et al. FORMULA9; BID8, the RL AS problem is solved with a portfolio composed of online RL algorithms. The main limitation from these works relies on the fact that on-policy algorithms were used, which prevents them from sharing trajectories among algorithms . Meta-learning specifically for the eligibility trace parameter has also been studied in BID21. BID19 study the learning process of RL algorithms and selects the best one for learning faster on a new task. This work is related to batch AS.An intuitive way to solve the AS problem is to consider algorithms as arms in a multi-armed bandit setting. The bandit meta-algorithm selects the algorithm controlling the next trajectory ε and the objective function µ(ε) constitutes the reward of the bandit. The aim of prediction with expert advice is to minimise the regret against the best expert of a set of predefined experts. When the experts learn during time, their performances evolve and hence the sequence of expert rewards is non-stationary. The exponential weight algorithms BID3 ) are designed for prediction with expert advice when the sequence of rewards of experts is generated by an oblivious adversary. This approach has been extended for competing against the best sequence of experts by adding in the update of weights a forgetting factor proportional to the mean reward (see Exp3.S in BID3), or by combining Exp3 with a concept drift detector BID0.The exponential weight algorithms have been extended to the case where the rewards are generated by any sequence of stochastic processes of unknown means .A recent article uses Exp3.S algorithm BID3 for curriculum learning. The drawback of adversarial approaches is that they lead to very conservative algorithms which has to work against an adversary. For handeling non-stationarity of rewards, another way is to assume that the rewards generated by each arm are not i.i.d., but are governed by some more complex stochastic processes. The stochastic bandit algorithm such as UCB can be extended to the case of switching bandits using a discount factor or a window to forget the past. Restless bandits BID22; BID13 assume that a Markov chain governs the reward of arms independently of whether the learner is played or not the arm. These classes of bandit algorithms are not designed for experts that learn and hence evolve at each time step. Our approach takes the opposite view of adversarial bandits: we design a stochastic algorithm specifically for curriculum learning based on the doubling trick. This reduction of the algorithm selection problem into several stochastic bandit problems with doubling time horizon begins to favour fast algorithms, and then more efficient algorithms. ESBAS description -To solve the off-policy RL AS problem, we propose a novel meta-algorithm called Epochal Stochastic Bandit AS (ESBAS). Because of the non-stationarity induced by the algorithm learning, the stochastic bandit cannot directly select algorithms. Instead, the stochastic bandit can choose fixed policies. To comply with this constraint, the meta-time scale is divided into epochs inside which the algorithms policies cannot be updated: the algorithms optimise their policies only when epochs start, in such a way that the policies are constant inside each epoch. This can be seen as a doubling trick. As a consequence and since the returns are bounded, at each new epoch, the problem can rigorously be cast into an independent stochastic K-armed bandit Ξ, with K = |P|. Data: D 0, P, µ: the online RL AS setting DISPLAYFORM0 n kmax + 1 n kmax ← n kmax + 1 and n ← n + 1 end endThe ESBAS meta-algorithm is formally sketched in Pseudo-code 2 embedding UCB1 Auer et al. (2002a) as the stochastic Karmed bandit Ξ. The meta-algorithm takes as an input the set of algorithms in the portfolio. Meta-time scale is fragmented into epochs of exponential size. The β th epoch lasts 2 β metatime steps, so that, at meta-time τ = 2 β, epoch β starts. At the beginning of each epoch, the ESBAS meta-algorithm asks each algorithm in the portfolio to update their current policy. Inside an epoch, the policy is never updated anymore. At the beginning of each epoch, a new Ξ instance is reset and run. During the whole epoch, Ξ selects at each meta-time step the algorithm in control of the next trajectory. Theoretical analysis -ESBAS intends to minimise the regret for not choosing the algorithm yielding the maximal return at a given meta-time τ. It is short-sighted: it does not intend to optimise the algorithms learning. We define the short-sighted pseudo-regret as follows: DISPLAYFORM1 The short-sighted pseudo-regret depends on the gaps ∆ α β: the difference of expected return between the best algorithm during epoch β and algorithm α. The smallest non null gap at epoch β is noted ∆ † β. We write its limit when β tends to infinity with ∆ † ∞. Analysis relies on three assumptions that are formalised in Section B of the supplementary material. First, more data is better data states that algorithms improve on average from having additional data. Second, order compatibility assumes that, if a dataset enables to generate a better policy than another dataset, then, on average, adding new samples to both datasets should not change the dataset ordering. Third and last, let us introduce and discuss more in depth the learning is fair assumption. The fairness of budget distribution has been formalised in. It is the property stating that every algorithm in the portfolio has as much resources as the others, in terms of computational time and data. It is an issue in most online AS problems, since the algorithm that has been the most selected has the most data, and therefore must be the most advanced one. A way to circumvent this issue is to select them equally, but, in an online setting, the goal of AS is precisely to select the best algorithm as often as possible. Our answer is to require that all algorithms in the portfolio are learning off-policy, i.e. without bias induced by the behavioural policy used in the learning dataset. By assuming that all algorithms learn off-policy, we allow information sharing Cauwet et al. FORMULA9 between algorithms. They share the trajectories they generate. As a consequence, we can assume that every algorithm, the least or the most selected ones, will learn from the same trajectory set. Therefore, the control unbalance does not directly lead to unfairness in algorithms performances: all algorithms learn equally from all trajectories. However, unbalance might still remain in the exploration strategy if, for instance, an algorithm takes more benefit from the exploration it has chosen than the one chosen by another algorithm. For analysis purposes, we assumes the complete fairness of AS.Based on those assumptions, three theorems show that ESBAS absolute pseudo-regret can be expressed in function of the absolute pseudo-regret of the best canonical algorithm and ESBAS shortsighted pseudo-regret. They also provide upper bounds on the ESBAS short-sighted pseudo-regret as a function of the order of magnitude of the gap ∆ † β. Indeed, the stochastic multi-armed bandit algorithms have bounds that are, counter-intuitively, inversely proportional to the gaps between the best arm and the other ones. In particular if ∆ † β tends to 0, the algorithm selection might prove to be difficult, depending on the order of magnitude of it tending to 0. The full theoretical analysis can be found in the supplementary material, Section B. We provide here an intuitive overlook of its . TAB0 numerically reports those bounds for a two-fold portfolio, depending on the nature of the algorithms. It must be read by line. According to the first column: the order of magnitude of ∆ † β, the ESBAS short-sighted pseudo-regret bounds are displayed in the second column, and the third and fourth columns display the ESBAS absolute pseudo-regret bounds also depending on the order of magnitude of the best canonical algorithm absolute pseudo-regret: ρ σ * Regarding the short-sighted upper bounds, the main appears in the last line, when the algorithms converge to policies with different performance: ESBAS converges with a regret in O log 2 (T)/∆ † ∞. Also, one should notice that the bounds of the first two lines are obtained by summing the gaps: this means that the algorithms are perceived equally good and that their gap goes beyond the threshold of distinguishability. This threshold is structurally at ∆ † DISPLAYFORM0 The impossibility to determine which is the better algorithm is interpreted in Cauwet et al. FORMULA9 as a budget issue. The meta-time necessary to distinguish through evaluation arms that are ∆ † β apart takes Θ(1/∆ †2 β) meta-time steps. If the budget is inferior, then we are under the distinguishability threshold and the best bounds are obtained by summing up the gaps. As a consequence, if ∆ † DISPLAYFORM1. However, the budget, i.e. the length of epoch β starting at meta-time T = 2 β, equals DISPLAYFORM2 can therefore be considered as the structural limit of distinguishability between the algorithms. Additionally, the absolute upper bounds are logarithmic in the best case and still inferior to O(√ T) in the worst case, which compares favorably with those of discounted UCB and Exp3.S in O(T log(T)) and Rexp3 in O(T 2/3), or the RL with Policy Advice's regret bounds of O(√ T log(T)) on stationary policies Azar et al. FORMULA9 (on non-episodic RL tasks). DISPLAYFORM3, and c DISPLAYFORM4 ESBAS is particularly designed for RL tasks when it is impossible to update the policy after every transition or episode. Policy update is very costly in most real-world applications, such as dialogue systems for which a growing batch setting is preferred BID6. ESBAS practical efficiency is therefore illustrated on a dialogue negotiation game BID7 ) that involves two players: the system p s and a user p u. Their goal is to find an agreement among 4 alternative options. At each dialogue, for each option η, players have a private uniformly drawn cost ν Table 2 of the supplementary material. Figures 3a and 3b plot the typical curves obtained with ESBAS selecting from a portfolio of two learning algorithms. On Figure 3a, the ESBAS curve tends to reach more or less the best algorithm in each point as expected. Surprisingly, Figure 3b reveals that the algorithm selection ratios are not very strong in favour of one or another at any time. Indeed, the variance in trajectory set collection makes simple better on some runs until the end. ESBAS proves to be efficient at selecting the best algorithm for each run and unexpectedly obtains a negative relative pseudo-regret of -90. Figures 3c and 3d plot the typical curves obtained with ESBAS selecting from a portfolio constituted of a learning algorithm and an algorithm with a deterministic and stationary policy. ESBAS succeeds in remaining close to the best algorithm at each epoch and saves 5361 return value for not selecting the constant algorithm, but overall yields a regret for not using only the best algorithm. ESBAS also performs well on larger portfolios of 8 learners (see Figure 3e) with negative relative pseudo-regrets: −10, even if the algorithms are, on average, almost selected uniformly as Figure 3f reveals. Each individual run may present different ratios, depending on the quality of the trained policies. ESBAS also offers some curriculum learning, but more importantly, early bad policies are avoided. Algorithms with a constant policy do not improve over time and the full reset of the K-multi armed bandit urges ESBAS to unnecessarily explore again and again the same underachieving algorithm. One easy way to circumvent this drawback is to use this knowledge and to not reset their arms. By operating this way, when the learning algorithm(s) start(s) outperforming the constant one, ESBAS simply neither exploits nor explores the constant algorithm anymore. Without arm reset for constant algorithms, ESBAS's learning curve follows perfectly the learning algorithm's learning curve when this one outperforms the constant algorithm and achieves strong negative relative pseudo-regrets. Again, the interested reader may refer to Table 2 in supplementary material for the numerical . Still, another harmful phenomenon may happen: the constant algorithm overrides the natural exploration of the learning algorithm in the early stages, and when the learning algorithm finally outperforms the constant algorithm, its exploration parameter is already low. This can be observed in experiments with constant algorithm of expected return inferior to 1, as reported in Table 2. We propose to adapt ESBAS to a true online setting where algorithms update their policies after each transition. The stochastic bandit is now trained on a sliding window containing the last τ /2 selections. Even though the arms are not stationary over this window, the bandit eventually forgets the oldest arm pulls. This algorithm is called SSBAS for Sliding Stochastic Bandit AS. Despite the lack of theoretical convergence bounds, we demonstrate on two domains and two different meta-optimisation tasks that SSBAS impressively outperforming all algorithms in the portfolio by a wide margin. The goal here is to demonstrate that SSBAS can perform efficient hyper-parameter optimisation on a simple tabular domain: a 5x5 gridworld problem (see Figure 4), where the goal is to collect the fruits placed at each corner as fast as possible. The episodes terminate when all fruits have been collected or after 100 transitions. The objective function µ used to optimise the stochastic bandit Ψ is no longer the RL return, but the time spent to collect all the fruits (200 in case of it did not). The agent has 18 possible positions and there are 2 4 − 1 = 15 non-terminal fruits configurations, ing in 270 states. The action set is A = {N, E, S, W}. The reward function mean is 1 when eating a fruit, 0 otherwise. The reward function is corrupted with a strong Gaussian white noise of variance ζ 2 = 1. The portfolio is composed of 4 Q-learning algorithms varying from each other by their learning rates: {0.001, 0.01, 0.1, 0.5}. They all have the same linearly annealing τ -greedy exploration. The selection ratios displayed in 5 show that SSBAS selected the algorithm with the highest (0.5) learning rate in the first stages, enabling to propagate efficiently the reward signal through the visited states, then, over time preferentially chooses the algorithm with a learning rate of 0.01, which is less sensible to the reward noise, finally, SSBAS favours the algorithm with the finest learning rate (0.001). After 1 million episodes, SSBAS enables to save half a transition per episode on average as compared to the best fixed learning rate value (0.1), and two transitions against the worst fixed learning rate in the portfolio (0.001).We compare SSBAS to the efficiency of a linearly annealing learning rate: 1/(1 + 0.0001τ): SSBAS performs under 21 steps on average after 10 5, while the linearly annealing learning rate algorithm still performs a bit over 21 steps after 10 6 steps. This is because SSBAS can adapt the best performing learning rate over time. We also compare SSBAS performance to Exp3.S's performance BID3. The analysis of the algorithm selection history shows that Exp3.S is too conservative and fails at efficiently select the shallowest algorithms in the beginning of the learning (number of steps at the 10000 th episode: 28.3 for SSBAS vs 39.1 for Exp3.S), producing trajectories of lesser quality and therefore critically delaying the general training of all algorithms (number of steps at the 100000 th episode: 20.9 for SSBAS vs 22.5 for Exp3.S). Overall, SSBAS outperforms Exp3.S by a significant and wide margin: number of steps averaged over all the training 10^5 episodes: 28.7 for SSBAS vs 33.6 for Exp3.S. FORMULA9 ) and more precisely the game Q*bert (see a frame on Figure 6), where the goal is to step once on each block. Then a new similar level starts. In later levels, one needs to step twice on each block, and even later stepping again on the same blocks will cancel the colour change. We used three different settings of DQN instances: small uses the setting described in BID10, large uses the setting in BID11, and finally huge uses an even larger network (see the supplementary material, Section C.2 for details). DQN is known to reach a near-human level performance at Q*bert. Our SSBAS instance runs 6 algorithms with 2 different random initialisations of each DQN setting. Disclaimer: contrarily to other experiments, each curve is the of a single run, and the improvement might be aleatory. Indeed, the DQN training is very long and SSBAS needs to train all the models in parallel. A more computationally-efficient solution might be to use the same architecture as BID14.7 reveals that SSBAS experiences a slight delay keeping in touch with the best setting performance during the initial learning phase, but, surprisingly, finds a better policy than the single algorithms in its portfolio and than the ones reported in the previous DQN articles. We observe that the large setting is surprisingly by far the worst one on the Q*bert task, implying the difficulty to predict which model is the most efficient for a new task. SSBAS allows to select online the best one. In this article, we tackle the problem of selecting online off-policy RL algorithms. The problem is formalised as follows: from a fixed portfolio of algorithms, a meta-algorithm learns which one performs the best on the task at hand. Fairness of algorithm evaluation implies that the RL algorithms learn off-policy. ESBAS, a novel meta-algorithm, is proposed. Its principle is to divide the meta-time scale into epochs. Algorithms are allowed to update their policies only at the start each epoch. As the policies are constant inside each epoch, the problem can be cast into a stochastic multi-armed bandit. An implementation is detailed and a theoretical analysis leads to upper bounds on the regrets. ESBAS is designed for the growing batch RL setting. This limited online setting is required in many real-world applications where updating the policy requires a lot of resources. Experiments are first led on a negotiation dialogue game, interacting with a human data-built simulated user. In most settings, not only ESBAS demonstrates its efficiency to select the best algorithm, but it also outperforms the best algorithm in the portfolio thanks to curriculum learning, and variance reduction similar to that of Ensemble Learning. Then, ESBAS is adapted to a full online setting, where algorithms are allowed to update after each transition. This meta-algorithm, called SSBAS, is empirically validated on a fruit collection task where it performs efficient hyper-parameter optimisation. SSBAS is also evaluated on the Q*bert Atari game, where it achieves a substantial improvement over the single algorithm counterparts. We interpret ESBAS/SSBAS's success at reliably outperforming the best algorithm in the portfolio as the of the four following potential added values. First, curriculum learning: ESBAS/SSBAS selects the algorithm that is the most fitted with the data size. This property allows for instance to use shallow algorithms when having only a few data and deep algorithms once collected a lot. Second, diversified policies: ESBAS/SSBAS computes and experiments several policies. Those diversified policies generate trajectories that are less redundant, and therefore more informational. As a , the policies trained on these trajectories should be more efficient. Third, robustness: if one algorithm fails at finding good policies, it will soon be discarded. This property prevents the agent from repeating again and again the same obvious mistakes. Four and last, run adaptation: of course, there has to be an algorithm that is the best on average for one given task at one given meta-time. But depending on the variance in the trajectory collection, it did not necessarily train the best policy for each run. The ESBAS/SSBAS meta-algorithm tries and selects the algorithm that is the best at each run. Some of those properties are inherited by algorithm selection similarity with ensemble learning . BID23 uses a vote amongst the algorithms to decide the control of the next transition. Instead, ESBAS/SSBAS selects the best performing algorithm. Regarding the portfolio design, it mostly depends on the available computational power per sample ratio. For practical implementations, we recommend to limit the use of two highly demanding algorithms, paired with several faster algorithms that can take care of first learning stages, and to use algorithms that are diverse regarding models, hypotheses, etc. Adding two algorithms that are too similar adds inertia, while they are likely to not be distinguishable by ESBAS/SSBAS. More detailed recommendations for building an efficient RL portfolio are left for future work. Speech recognition score Section C.1.1 Normal distribution of centre x and variance v 2 Section C.1.1 REFINSIST REFPROP(η), with η being the last proposed option Section C.1.1 REFNEWPROP REFPROP(η), with η being the best option that has not been proposed yet Section C.1.1 ACCEPT ACCEPT(η), with η being the last understood option proposition Section C. DISPLAYFORM0 Non-learning algorithm with average performance µ Section C.1.2 ζ Number of noisy features added to the feature set Section C.1.2 Probability that X = x conditionally to Y = y Equation FORMULA11 B THEORETICAL ANALYSISThe theoretical aspects of algorithm selection for reinforcement learning in general, and Epochal Stochastic Bandit Algorithm Selection in particular, are thoroughly detailed in this section. The proofs of the Theorems are provided in Sections E, F, and G. We recall and formalise the absolute pseudo-regret definition provided in Section 2.3.Definition 1 (Absolute pseudo-regret). The absolute pseudo-regret ρ σ abs (T) compares the metaalgorithm's expected return with the optimal expected return: DISPLAYFORM0 The theoretical analysis is hindered by the fact that AS, not only directly influences the return distribution, but also the trajectory set distribution and therefore the policies learnt by algorithms for next trajectories, which will indirectly affect the future expected returns. In order to allow policy comparison, based on relation on trajectory sets they are derived from, our analysis relies on two assumptions. Assumption 1 (More data is better data). The algorithms train better policies with a larger trajectory set on average, whatever the algorithm that controlled the additional trajectory: DISPLAYFORM0 Assumption 1 states that algorithms are off-policy learners and that additional data cannot lead to performance degradation on average. An algorithm that is not off-policy could be biased by a specific behavioural policy and would therefore transgress this assumption. If an algorithm trains a better policy with one trajectory set than with another, then it remains the same, on average, after collecting an additional trajectory from any algorithm: DISPLAYFORM0 Assumption 2 states that a performance relation between two policies trained on two trajectory sets is preserved on average after adding another trajectory, whatever the behavioural policy used to generate it. From these two assumptions, Theorem 1 provides an upper bound in order of magnitude in function of the worst algorithm in the portfolio. It is verified for any meta-algorithm σ. Theorem 1 (Not worse than the worst). The absolute pseudo-regret is bounded by the worst algorithm absolute pseudo-regret in order of magnitude: DISPLAYFORM1 Contrarily to what the name of Theorem 1 suggests, a meta-algorithm might be worse than the worst algorithm (similarly, it can be better than the best algorithm), but not in order of magnitude. Its proof is rather complex for such an intuitive because, in order to control all the possible outcomes, one needs to translate the selections of algorithm α with meta-algorithm σ into the canonical meta-algorithm σ α's view. ESBAS intends to minimise the regret for not choosing the best algorithm at a given meta-time τ. It is short-sighted: it does not intend to optimise the algorithms learning. Definition 2 (Short-sighted pseudo-regret). The short-sighted pseudo-regret ρ σ ss (T) is the difference between the immediate best expected return algorithm and the one selected: DISPLAYFORM0 Theorem 2 (ESBAS short-sighted pseudo-regret). If the stochastic multi-armed bandit Ξ guarantees a regret of order of magnitude O(log(T)/∆ † β ), then: DISPLAYFORM1 Theorem 2 expresses in order of magnitude an upper bound for the short-sighted pseudo-regret of ESBAS. But first, let define the gaps: DISPLAYFORM2. It is the difference of expected return between the best algorithm during epoch β and algorithm α. The smallest non null gap at epoch β is noted: DISPLAYFORM3 if there is no non-null gap, the regret is null. Several upper bounds in order of magnitude on ρ ss (T) can be easily deduced from Theorem 2, depending on an order of magnitude of ∆ † β. See the corollaries in Section F.1, TAB0 and more generally Section 3 for a discussion. The short-sighted pseudo-regret optimality depends on the meta-algorithm itself. For instance, a poor deterministic algorithm might be optimal at meta-time τ but yield no new information, implying the same situation at meta-time τ + 1, and so on. Thus, a meta-algorithm that exclusively selects the deterministic algorithm would achieve a short-sighted pseudo-regret equal to 0, but selecting other algorithms are, in the long run, more efficient. Theorem 2 is a necessary step towards the absolute pseudo-regret analysis. The absolute pseudo-regret can be decomposed between the absolute pseudo-regret of the best canonical meta-algorithm (i.e. the algorithm that finds the best policy), the regret for not always selecting the best algorithm, and potentially not learning as fast, and the short-sighted regret: the regret for not gaining the returns granted by the best algorithm. This decomposition leads to Theorem 3 that provides an upper bound of the absolute pseudo-regret in function of the best canonical meta-algorithm, and the short-sighted pseudo-regret. The fairness of budget distribution is the property stating that every algorithm in the portfolio has as much resources as the others, in terms of computational time and data. Section 3 discusses it at length. For analysis purposes, Theorem 3 assumes the fairness of AS:Assumption 3 (Learning is fair). If one trajectory set is better than another for training one given algorithm, it is the same for other algorithms. DISPLAYFORM0 Theorem 3 (ESBAS absolute pseudo-regret upper bound). Under assumption 3, if the stochastic multi-armed bandit Ξ guarantees that the best arm has been selected in the T first episodes at least T /K times, with high probability 1 − δ T, with δ T ∈ O(1/T), then: DISPLAYFORM1 where meta-algorithm σ * selects exclusively algorithm α * = argmin α∈P ρ σ α abs (T). Successive and Median Elimination and Upper Confidence Bound BID2 under some conditions BID1 are examples of appropriate Ξ satisfying both conditions stated in Theorems 2 and 3. Again, see TAB0 and more generally Section 3 for a discussion of those bounds. C EXPERIMENTAL DETAILS C.1 DIALOGUE EXPERIMENTS DETAILS C.1.1 THE NEGOTIATION DIALOGUE GAME ESBAS practical efficiency is illustrated on a dialogue negotiation game BID7 that involves two players: the system p s and a user p u. Their goal is to find an agreement among 4 alternative options. At each dialogue, for each option η, players have a private uniformly drawn cost ν p η ∼ U to agree on it. Each player is considered fully empathetic to the other one. As a , if the players come to an agreement, the system's immediate reward at the end of the dialogue is R ps (s f) = 2 − ν ps η − ν pu η, where s f is the state reached by player p s at the end of the dialogue, and η is the agreed option; if the players fail to agree, the final immediate reward is R ps (s f) = 0, and finally, if one player misunderstands and agrees on a wrong option, the system gets the cost of selecting option η without the reward of successfully reaching an agreement: R ps (s f) = −ν ps η − ν pu η. Players act each one in turn, starting randomly by one or the other. They have four possible actions. First, REFPROP(η): the player makes a proposition: option η. If there was any option previously proposed by the other player, the player refuses it. Second, ASKREPEAT: the player asks the other player to repeat its proposition. Third, ACCEPT(η): the player accepts option η that was understood to be proposed by the other player. This act ends the dialogue either way: whether the understood proposition was the right one or not. Four, ENDDIAL: the player does not want to negotiate anymore and ends the dialogue with a null reward. Understanding through speech recognition of system p s is assumed to be noisy: with a sentence error rate of probability SER u s = 0.3, an error is made, and the system understands a random option instead of the one that was actually pronounced. In order to reflect human-machine dialogue asymmetry, the simulated user always understands what the system says: SER The system, and therefore the portfolio algorithms, have their action set restrained to five non parametric actions: REFINSIST ⇔ REFPROP(η t−1), η t−1 being the option lastly proposed by the system; REFNEWPROP ⇔ REFPROP(η), η being the preferred one after η t−1, ASKREPEAT, ACCEPT⇔ ACCEPT(η), η being the last understood option proposition and ENDDIAL. All learning algorithms are using Fitted-Q Iteration , with a linear parametrisation and an β -greedy exploration: β = 0.6 β, β being the epoch number. Six algorithms differing by their state space representation Φ α are considered:• simple: state space representation of four features: the constant feature φ 0 = 1, the last recognition score feature φ asr, the difference between the cost of the proposed option and the next best option φ dif, and finally an RL-time feature φ t = 0.1t DISPLAYFORM0 • fast: Φ α = {φ 0, φ asr, φ dif}.• simple-2: state space representation of ten second order polynomials of simple features. DISPLAYFORM1 • fast-2: state space representation of six second order polynomials of fast features. DISPLAYFORM2 • n-ζ-{simple/fast/simple-2/fast-2}: Versions of previous algorithms with ζ additional features of noise, randomly drawn from the uniform distribution in.• constant-µ: the algorithm follows a deterministic policy of average performance µ without exploration nor learning. Those constant policies are generated with simple-2 learning from a predefined batch of limited size. In all our experiments, ESBAS has been run with UCB parameter ξ = 1/4. We consider 12 epochs. The first and second epochs last 20 meta-time steps, then their lengths double at each new epoch, for a total of 40,920 meta-time steps and as many trajectories. γ is set to 0.9. The algorithms and ESBAS are playing with a stationary user simulator built through Imitation Learning from realhuman data. All the are averaged over 1000 runs. The performance figures plot the curves of algorithms individual performance σ α against the ESBAS portfolio control σ ESBAS in function of the epoch (the scale is therefore logarithmic in meta-time). The performance is the average return of the reinforcement learning return: it equals γ | | R ps (s f) in the negotiation game. The ratio figures plot the average algorithm selection proportions of ESBAS at each epoch. We define the relative pseudo regret as the difference between the ESBAS absolute pseudo-regret and the absolute pseudo-regret of the best canonical meta-algorithm. All relative pseudo-regrets, as well as the gain for not having chosen the worst algorithm in the portfolio, are provided in Table 2. Relative pseudo-regrets have a 95% confidence interval about ±6 ≈ ±1.5 × 10 −4per trajectory. Several show that, in practice, the assumptions are transgressed. Firstly, we observe that Assumption 3 is transgressed. Indeed, it states that if a trajectory set is better than another for a given algorithm, then it's the same for the other algorithms. Still, this assumption infringement does not seem to harm the experimental . It even seems to help in general: while this assumption is consistent curriculum learning, it is inconsistent with the run adaptation property advanced in Subsection 6 that states that an algorithm might be the best on some run and another one on other runs. And secondly, off-policy reinforcement learning algorithms exist, but in practice, we use state space representations that distort their off-policy property. However, experiments do not reveal any obvious bias related to the off/on-policiness of the trajectory set the algorithms train on. The three DQN networks (small, large, and huge) are built in a similar fashion, with relu activations at each layer except for the output layer which is linear, with RMSprop optimizer (ρ = 0.95 and = 10 no-op max 30• small has a first convolution layer with a 4x4 kernel and a 2x2 stride, and a second convolution layer with a 4x4 kernel and a 2x2 stride, followed by a dense layer of size 128, and finally the output layer is also dense.• large has a first convolution layer with a 8x8 kernel and a 4x4 stride, and a second convolution layer with a 4x4 kernel and a 2x2 stride, followed by a dense layer of size 256, and finally the output layer is also dense.• huge has a first convolution layer with a 8x8 kernel and a 4x4 stride, a second convolution layer with a 4x4 kernel and a 2x2 stride, and a third convolution layer with a 3x3 kernel and a 1x1 stride, followed by a dense layer of size 512, and finally the output layer is also dense. Portfolio w. best w. worst simple-2 + fast-2 35 -181 simple + n-1-simple-2 -73 -131 simple + n-1-simple 3 -2 simple-2 + n-1-simple-2 -12 -38 all-4 + constant-1.10 21 -2032 all-4 + constant-1.11-21 -1414 all-4 + constant-1.13-10 -561 all-4-28 -275 all-2-simple + constant-1.08-41 -2734 all-2-simple + constant-1.11-40 -2013 all-2-simple + constant-1.13-123 -799 all-2-simple -90 -121 fast + simple-2 -39 -256 simple-2 + constant-1.01 169 -5361 simple-2 + constant-1.11 53 -1380 simple-2 + constant-1.11 57 -1288 simple + constant-1.08 54 -2622 simple + constant-1.10 88 -1565 simple + constant-1.14 -6 -297 all-4 + all-4-n-1 + constant-1.09 25 -2308 all-4 + all-4-n-1 + constant-1.11 20 -1324 all-4 + all-4-n-1 + constant-1.14 -16 -348 all-4 + all-4-n-1 -10 -142 all-2-simple + all-2-n-1-simple -80 -181 4*n-2-simple -20 -20 4*n-3-simple -13 -13 8*n-1-simple-2 -22 -22 simple-2 + constant-0.97 (no reset) 113 -7131 simple-2 + constant-1.05 (no reset) 23 -3756 simple-2 + constant-1.09 (no reset) -19 -2170 simple-2 + constant-1.13 (no reset) -16 -703 simple-2 + constant-1.14 (no reset) -125 -319 Table 2: ESBAS pseudo-regret after 12 epochs (i.e. 40,920 trajectories) compared with the best and the worst algorithms in the portfolio, in function of the algorithms in the portfolio (described in the first column). The'+' character is used to separate the algorithms. all-4 means all the four learning algorithms described in Section C.1.2 simple + fast + simple-2 + fast-2. all-4-n-1 means the same four algorithms with one additional feature of noise. Finally, all-2-simple means simple + simple-2 and all-2-n-1-simple means n-1-simple + n-1-simple-2. On the second column, the redder the colour, the worse ESBAS is achieving in comparison with the best algorithm. Inversely, the greener the colour of the number, the better ESBAS is achieving in comparison with the best algorithm. If the number is neither red nor green, it means that the difference between the portfolio and the best algorithm is insignificant and that they are performing as good. This is already an achievement for ESBAS to be as good as the best. On the third column, the bluer the cell, the weaker is the worst algorithm in the portfolio. One can notice that positive regrets are always triggered by a very weak worst algorithm in the portfolio. In these cases, ESBAS did not allow to outperform the best algorithm in the portfolio, but it can still be credited with the fact it dismissed efficiently the very weak algorithms in the portfolio. Theorem 1 (Not worse than the worst). The absolute pseudo-regret is bounded by the worst algorithm absolute pseudo-regret in order of magnitude: DISPLAYFORM0 Proof. From Definition 1: DISPLAYFORM1 where sub α (D) is the subset of D with all the trajectories generated with algorithm α, where τ α i is the index of the i th trajectory generated with algorithm α, and where |S| is the cardinality of finite set S. By convention, let us state that Eµ DISPLAYFORM2 To conclude, let us prove by mathematical induction the following inequality: DISPLAYFORM3 is true by vacuity for i = 0: both left and right terms equal Eµ α ∅. Now let us assume the property true for i and prove it for i + 1: DISPLAYFORM4.If |sub α (D σ T)|≥ i + 1, by applying mathematical induction assumption, then by applying Assumption 2 and finally by applying Assumption 1 recursively, we infer that: The mathematical induction proof is complete. This leads to the following inequalities: DISPLAYFORM5 DISPLAYFORM6 which leads directly to the : DISPLAYFORM7 This proof may seem to the reader rather complex for such an intuitive and loose but algorithm selection σ and the algorithms it selects may act tricky. For instance selecting algorithm α only when the collected trajectory sets contains misleading examples (i.e. with worse expected return than with an empty trajectory set) implies that the following unintuitive inequality is always true: Eµ DISPLAYFORM8. In order to control all the possible outcomes, one needs to translate the selections of algorithm α into σ α's view. Theorem 2 (ESBAS short-sighted pseudo-regret). If the stochastic multi-armed bandit Ξ guarantees a regret of order of magnitude O(log(T)/∆ † β ), then: DISPLAYFORM0 Proof. By simplification of notation, Eµ DISPLAYFORM1. From Definition 2: Since we are interested in the order of magnitude, we can once again only consider the upper bound of DISPLAYFORM2 DISPLAYFORM3 Theorem 3 (ESBAS absolute pseudo-regret upper bound). Under assumption 3, if the stochastic multi-armed bandit Ξ guarantees that the best arm has been selected in the T first episodes at least T /K times, with high probability 1 − δ T, with δ T ∈ O(1/T), then: DISPLAYFORM0 where meta-algorithm σ * selects exclusively algorithm α * = argmin α∈P ρ σ α abs (T).Proof. The ESBAS absolute pseudo-regret is written with the following notation simplifications: DISPLAYFORM1 and k τ = σ ESBAS (τ): Note that σ * is the optimal constant algorithm selection at horizon T, but it is not necessarily the optimal algorithm selection: there might exist, and there probably exists a non constant algorithm selection yielding a smaller pseudo-regret. DISPLAYFORM2 The ESBAS absolute pseudo-regret ρ σ ESBAS abs (T) can be decomposed into the pseudo-regret for not having followed the optimal constant algorithm selection σ * and the pseudo-regret for not having selected the algorithm with the highest return, i.e. between the pseudo-regret on the trajectory and the pseudo-regret on the immediate optimal return:. is to evaluate the size of sub * (D τ −1).On the one side, Assumption 3 of fairness states that one algorithm learns as fast as any another over any history. The asymptotically optimal algorithm(s) when τ → ∞ is(are) therefore the same one(s) whatever the the algorithm selection is. On the other side, let 1 − δ τ denote the probability, that at time τ, the following inequality is true: DISPLAYFORM0 With probability δ τ, inequality 34 is not guaranteed and nothing can be inferred about Eµ * sub * (Dτ−1), except it is bounded under by R min /(1 − γ). Let E DISPLAYFORM1 Let consider E * (α, N) the set of all sets D such that |sub α (D)|= N and such that last trajectory in D was generated by α. Since ESBAS, with Ξ, a stochastic bandit with regret in O(log(T)/∆), guarantees that all algorithms will eventually be selected an infinity of times, we know that: ∀α ∈ P, ∀N ∈ N, D ∈E + (α,N) P(D|σ ESBAS) = 1.By applying recursively Assumption 2, one demonstrates that: DISPLAYFORM2 D ∈E + (α,N) DISPLAYFORM3 One also notices the following piece-wisely domination from applying recursively Assumption 1: DISPLAYFORM4 (1 − δ τ)Eµ * ∞ − D ∈E; the meta-time spent on epoch β τ − 1 is equal to 2 βτ −1; the meta-time spent on epoch β is either below 2 βτ −1, in which case, the meta-time spent on epoch β τ − 1 is higher than τ 3, or the meta-time spent on epoch β is over 2 βτ −1 and therefore higher than τ 3. Thus, ESBAS is guaranteed to try the best algorithm α * at least τ /3K times with high probability 1 − δ τ and δ τ ∈ O(τ −1). As a : DISPLAYFORM5 with κ = κ 3 Eµ * ∞ − R min 1 − γ, which proves the theorem.

This paper formalises the problem of online algorithm selection in the context of Reinforcement Learning.

Represent the short summary for finding the corresponding scientific article (long)

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.