Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: In high-dimensional reinforcement learning settings with sparse rewards, performing effective exploration to even obtain any reward signal is an open challenge. While model-based approaches hold promise of better exploration via planning, it is extremely difficult to learn a reliable enough Markov Decision Process (MDP) in high dimensions (e.g., over 10^100 states). In this paper, we propose learning an abstract MDP over a much smaller number of states (e.g., 10^5), which we can plan over for effective exploration. We assume we have an abstraction function that maps concrete states (e.g., raw pixels) to abstract states (e.g., agent position, ignoring other objects). In our approach, a manager maintains an abstract MDP over a subset of the abstract states, which grows monotonically through targeted exploration (possible due to the abstract MDP). Concurrently, we learn a worker policy to travel between abstract states; the worker deals with the messiness of concrete states and presents a clean abstraction to the manager. On three of the hardest games from the Arcade Learning Environment (Montezuma's, Pitfall!, and Private Eye), our approach outperforms the previous state-of-the-art by over a factor of 2 in each game. In Pitfall!, our approach is the first to achieve superhuman performance without demonstrations. Exploration is a key bottleneck in high-dimensional, sparse-reward reinforcement learning tasks. Random exploration (e.g., via epsilon-greedy) suffices when rewards are abundant BID24, but when rewards are sparse, it can be difficult for an agent starting out to even find any positive reward needed to bootstrap learning. For example, the infamously difficult game MON-TEZUMA'S REVENGE from the Arcade Learning Environment (ALE) BID6 contains over 10 100 states and requires the agent to go thousands of timesteps without receiving reward. Performing effective exploration in this setting is thus an open problem; without demonstrations, even state-of-the-art intrinsically-motivated RL agents BID5 BID44 achieve only about one-tenth the score of an expert human.In this paper, we investigate model-based reinforcement learning BID17 as a potential solution to the exploration problem. The hope is that with a model of the state transitions and rewards, one can perform planning under the model to obtain a more informed exploration strategy. However, as the model being learned is imperfect, errors in the model compound BID42 BID43 when planning over many time steps. Furthermore, even if a perfect model were known, in high-dimensional state spaces (e.g. over 10 100 states), planning-computing the optimal policy (e.g. via value iteration)-is intractable. As a , model-based RL has had limited success in high-dimensional settings BID42. To address this, some prior work has focused on learning more accurate models by using more expressive function approximators BID25, and learning local models BID21 BID50. Others have attempted to robustly use imperfect models by conditioning on, instead of directly following, model-based rollouts BID49, frequently replanning, and combining model-based with model-free approaches BID0 BID40. However, none of these techniques offer a fundamental solution. Instead of directly learning a model over the concrete state space, we propose an approach inspired by hierarchical reinforcement learning (HRL) BID41 BID46 Figure 1: (a) Illustration of the abstract MDP on MONTEZUMA'S REVENGE. We have superimposed a white grid on top of the original game. At any given time, the agent is in one of the grid cellseach grid cell is an abstract state. In this example, the agent starts at the top of a ladder (yellow dot). The worker then navigates transitions between abstract states (green arrows) to follow a plan made by the manager (red dots). (b) Circles represent abstract states. Shaded circles represent states within the known set. The manager navigates the agent to the fringe of the known set (s 3), then randomly explores with π d to discover new transitions near s 3 (dotted box). (c) The worker extends the abstract MDP by learning to navigate to the newly discovered abstract states (dotted arrows). 2017), and learn a model over a much smaller abstract state space. Specifically, we assume we have a state abstraction function BID22 BID36 BID9, which maps a highdimensional concrete state (e.g. all pixels on the screen) to a low-dimensional abstract state (e.g. the position of the agent). We then aim to learn an (abstract) Markov Decision Process (MDP) over this abstract state space as follows: A manager maintains an (abstract) MDP over a subset of all possible abstract states which we call the known set, which is grown over time. The crucial property we enforce is that this abstract MDP is highly accurate and near deterministic on the known set, so we can perform planning without suffering from compounding errors, and do it efficiently since we are working with a much smaller number of abstract states. Concurrently, we learn a worker policy that the manager uses to transition between abstract states. The worker policy has access to the concrete states; its goal is to hide the messy details of the real world from the manager (e.g., jumping over monsters) so that the manager has a much simpler planning problem (e.g., traversing between two locations). In our implementation, the worker keeps an inventory of skills (i.e., options BID41), each of which is driven by a deep neural network; the worker assigns an appropriate skill for each transition between abstract states. In this way, the worker does not "forget" BID20, and we ensure monotonic progress in learning the abstract MDP. This abstract MDP, which enables us to efficiently explore via planning, is a key difference between our work and previous HRL work (e.g., BID4 BID46), which also learn skills and operate on latent abstract state spaces but without forming an MDP.We evaluate our approach on three of the most challenging games from the ALE BID6: MONTEZUMA'S REVENGE, PITFALL!, and PRIVATE EYE. In all three domains, our approach achieves more than 2x the reward of prior non-demonstration state-of-the-art approaches. In PITFALL!, we are the first to achieve superhuman performance without demonstrations, surpassing the prior state-of-the-art by over 100x. Additionally, since our approach is model-based, we can generalize to new rewards without re-training, as long as the reward function is a function of the abstract states. When evaluated on a new reward function never seen during training, our approach achieves over 3x the reward of prior state-of-the-art methods explicitly trained on the new rewards. We assume the world is an unknown episodic finite-horizon MDP with (concrete) states x ∈ X and actions a ∈ A. We further assume we have a simple predefined state abstraction function mapping concrete states x to abstract states s = φ(x). In MONTEZUMA'S REVENGE, for instance, a concrete state contains the pixels on the screen, while the corresponding abstract state contains the agent's position and inventory (Figure 1). We assume that reward only depends on the abstract states: taking any action transitioning from concrete state x to x leads to reward R(φ(x), φ(x)).Model-based approaches promise better exploration via planning, but struggle in high-dimensional state spaces due to compounding errors and computational intractability. To avoid these problems, we propose to construct and operate on a low-dimensional representation of the world consisting of abstract states, which we call the abstract MDP (we refer to the original MDP, the world, as the concrete MDP). Then we plan in the abstract MDP. Concretely, the abstract MDP consists of:• The state space is a subset of all abstract states, which we call the known set S, consisting of abstract states that can be reliably reached from the initial abstract state via actions in the action set. Over time, we monotonically grow the known set, which initially only contains the starting abstract state.• The action set comprises of calling the worker policy π w (a|x, (s, s)) on transitions (s, s) from the current abstract state s to a nearby abstract state s. When called on a transition (s, s), the worker navigates from s to s by taking concrete actions a conditioned on the current concrete state x. The worker abstracts away the messiness of the underlying concrete MDP so that all other parts of the system can operate on the abstract MDP by calling the worker. We denote calling the worker on transition (s, s) as the action go(s, s).• The transition dynamics of calling action go(s, s) at abstract state s are defined by the original MDP: i.e., if the worker takes a concrete trajectory x 0, a 0, x 1, a 1, · · ·, x T, the ing abstract state is φ(x T). The rewards for transitioning from s to s are the rewards in the concrete MDP R(s, s), which only depend on the abstract states by assumption. The core idea behind our approach is to construct the abstract MDP (Section 3) by growing the action set (training the worker), which in turn grows the known set. At each point in time, the manager maintains the known set and (accurate, to avoid compounding errors) estimates of the reward and transition dynamics of the abstract MDP. With these dynamics estimates, the manager can solve the abstract MDP at all times (e.g., by value iteration), since the abstract MDP is small. As the abstract MDP grows, it captures more and more of the concrete MDP, enabling the manager to recover a better and better policy via planning. Ultimately, the known set of the abstract MDP contains all abstract states, enabling the manager to recover a high-reward policy. As the manager constructs the abstract MDP, the abstract MDP maintains two key properties:• The action set of the abstract MDP consists of only reliable actions, actions go(s, s) that transition from abstract state s to abstract state s with probability at least 1 − δ for some small δ to avoid compounding uncertainty. This enables the manager to reach any abstract state in the known set with high probability. To simplify notation, the manager estimates the success rate P (s, s) of action go(s, s) instead of the full dynamics P (•|go(s, s), s), treating the (vanishingly small) fraction of failures equally.• The action set and known set of the abstract MDP grow monotonically. Since the action set only contains reliable transitions, a key danger is if learning new reliable transitions (adding new actions) causes the worker to forget already learned transitions (removing actions), stalling progress. We opt for a non-parametric approach, where the worker learns a skill (neural subpolicy) for each transition, reusing skills when possible. When a worker learns to reliably traverse a transition, it freezes the corresponding skill's parameters. The manager's goal is to fully construct the abstract MDP so that the known set contains all abstract states. Then, it can compute a high-reward policy on the concrete MDP via planning on the abstract MDP. To construct the abstract MDP, the manager adds new actions to the abstract MDP: training the worker to reliably traverse new transitions (driving the transition success rates toward 1). Concretely, the manager discovers new transitions, trains the worker on these transitions, and updates its dynamics estimates using Algorithm 1. On each episode, the manager either chooses to discover new transitions via randomized exploration (Section 3.1) or trains the worker. This is done by constructing a prioritized list of exploration goals, where each goal is either a transition to learn Score all candidates and select highest priority candidate c 5:Compute a plan (s0, s1), (s1, s2), · · ·, (sT −1, sT = s) with model 6:for t = 1 to T do 7:Call worker to navigate transition (st−1, st) 8:if c is a transition (s, s) to learn then 9:LEARNWORKER(s, s) 10:else c is an abstract state s to explore 11:DISCOVERTRANSITIONS or an abstract state to find nearby transitions (Section 3.2). Upon selecting the highest-priority goal (e.g., the transition (s, s)), the manager navigates to the relevant abstract state (e.g., s) by planning with its dynamics models (e.g., the plan go(s 0, s 1), go(s 1, s 2), · · ·, go(s T −1, s T = s)), executing the plan, and then calling the worker on or randomly exploring from the selected goal. Finally, the manager updates its dynamics models. It uses a sliding window estimate of the past N transition worker attempts of traversing transition (s, s) for the transition dynamics, and updates its reward estimate of a transition (s, s) as the reward accumulated by the first successful traversal of (s, s). When a transition (s, s) becomes reliable (i.e., the dynamics T (s, s) exceeds the threshold 1 − δ), the manager adds go(s, s) to the action set of the abstract MDP and adds s to the known set. For the manager to train the worker on new transitions, it must first discover new transitions. To discover new transitions, the manager navigates to an exploration candidate: an abstract state s. Then, it performs randomized exploration to discover new transitions (s, s) to nearby abstract states s. As exploration candidates, the manager simply chooses the abstract states in the known set that have been explored fewer than N visit times: i.e., n(s) < N visit, where n(s) is the number of times the manager has explored abstract state s for nearby transitions. Effectively, the manager assumes that when n(s) ≥ N visit, all nearby transitions (s, s) have been already discovered. Concretely, at an abstract state s, the manager finds nearby transitions by following a simple policy π d (a t |x 0:t, a 0:t−1) for T d timesteps (Algorithm 3). The policy π d outputs randomized concrete actions a t conditioned on the past concrete states x 0:t and past concrete actions a 0:t−1, where φ(x 0) is the abstract state s it was initially invoked. During those T d timesteps, the manager records the transitions and rewards it observes: (φ(x 0), r 0, φ(x 1)), · · · (φ(x T −1), r T −1, φ(x T)), using the rewards to update its rewards model and the transitions as candidates for the worker to learn. Additionally, if it ends in another exploration candidate (i.e., n(s) < N visit ) after exploring for T d timesteps, it simply continues exploring for another T d timesteps. The simplest possible policy for the π d is to uniformly sample a concrete action at each timestep. However, we found that this inadequately discovered new transitions, because it would often perform useless action sequences (e.g., left, right, left, right). Instead, we use a simple method for π d to commit to an exploration direction. At each timestep, the π d uniformly samples a concrete action and a number between 1 and T repeat, and repeats the action the sampled number of times. Exploration goals. The manager selects an exploration goal from the set of all candidate exploration goals, consisting of exploration candidates for the transition discovery and candidate transitions for the worker to learn. The exploration candidates are just the abstract states in the known set with n(s) < N transition. The candidate transitions are the transitions discovered by the manager. In addition, the worker imposes a heuristic on its transition learning process in order to preserve the Markov property of the abstract MDP (Section 4), which sometimes makes it impossible for the worker to learn a transition. To avoid getting stuck, as candidates for the worker, the manager also considers "long-distance" transitions: (s, s) pairs for which the manager did not directly transition from s to s, but indirectly did so through a sequence of intermediate states DISPLAYFORM0 Input: a transition (s, s) to learn, called at concrete state x0 with φ(x0) = s 1: Set worker horizon DISPLAYFORM1 Observe xt 5:Compute worker intrinsic reward rt = R (s,s) (xt|s) 6:Update worker on (xt−1, at−1, rt, xt) 7:Choose DISPLAYFORM2 Freeze worker's skill π I(s,s) DISPLAYFORM3 be the length of the shortest such path, the manager considers all pairs (s, DISPLAYFORM4 Priority scores of the exploration goals. The manager must choose exploration goals in some order. Our theoretical (Section 5) hold for any priority function that eventually chooses all exploration goals. In our implementation, the manager prioritizes the easiest exploration goals (enabling the fastest growth of the action set), and the most useful goals (goal that either lead to more reward or enable the worker to learn new transitions).Concretely, the manager heuristically computes the easiness of a learning transition (s, s) as DISPLAYFORM5 where n succ is the number of times the worker has successfully traversed (s, s) and n f ail is the number of times the worker has failed in traversing (s, s). Intuitively, both 1) succeeding more and failing less and 2) shorter transitions, requiring fewer timesteps indicate easier to learn transitions. Similarly, for an abstract state s, the manager computes the easiness of discovering new neighboring transitions as e(s) = −n(s) since abstract states that have been explored less are more likely to have undiscovered transitions. The manager heuristically computes the usefulness of a learning a transition (s, s) as u(s, s) = λ 2 I new + R(s 0, s), where I new is an indicator that is 1 if there is an outgoing transition (s, s) and no current candidate transitions end in s. R(s 0, s) is the reward achieved by navigating from the initial abstract state s 0 to s. If I new is 1, then learning (s, s) opens new candidate transitions for the worker to learn, indicating that learning (s, s) is useful. For an abstract state s, the manager computes the usefulness just as u(s) = λ 3 + R(s 0, s). The λ 3 constant accounts for how much more or less useful discovering new transitions is compared to learning new transitions. To prioritize exploration goals, the manager uniformly switches between two priority functions. The first priority function simply equals the easiness plus usefulness: e + u, and the second priority function is the same, but without the reward term in u, to avoid falling into a local reward maximum. The worker forms the action set of the abstract MDP by learning many subtasks of the form: navigate from abstract state s to s. It does this while maintaining the three properties: 1) the worker reliably (with high probability) traverses (s, s) for each action go(s, s) in the abstract MDP; 2) the action set grows monotonically, so learning new transitions never causes the worker's old transitions to become unreliable; and 3) the worker learns transitions in a way that preserves the Markov property. While it is possible to learn a single policy for all transition subtasks, it is tricky to satisfy 2), since learning new transitions can have deleterious effects on previously learned transitions. Instead, the worker maintains an inventory of skills (Section A.3), where each transition is learned by a single skill, sharing the same skill amongst many transitions when possible. The worker uses these skills to form the action set of the abstract MDP following Algorithm 2: When the manager calls the worker on a transition (s, s), the worker selects the appropriate skill from the skill inventory and begins an episode of the subtask of traversing s to s (Section 4.2). During the skill episode, the skill receives intrinsic rewards, and is declared to have successfully completed the subtask if it meets the worker's holding heuristic, which heuristically maintains 3) by ensuring the worker can always control the abstract state. If at the end of the skill episode, the success rate of the worker traversing (s, s) exceeds the reliability threshold 1 − δ, the action go(s, s) is added to the abstract MDP. The worker's skill inventory I indexes skills so that the skill at index I(s, s) reliably traverses transition (s, s). Each skill is a goal-conditioned subpolicy π I(s,s) (a|x, s), which produces concrete actions a conditioned on the current concrete state x and the goal abstract state s. When the worker traverses a transition (s, s), it calls on the corresponding skill until the transition is traversed: i.e., π w (a|x, (s, s)) = π I(s,s) (a|x, s).When learning a new transition (s, s), the worker first tries to reuse its already learned skills from the skill inventory. For each skill π i in the skill inventory, it measures the success rate of π i on the new transition (s, s) over N transition attempts. If the success rate exceeds the reliability threshold 1 − δ for any skill π i, it updates the skill repository to reuse the skill: I(s, s) ← π i. Otherwise, if no already learned skill can reliably traverse the new transition, the worker creates a new skill and trains it to navigate the transition by optimizing intrinsic rewards during skill episodes (Section 4.2). Given a transition (s, s), the worker's subtask is to navigate from abstract state s to abstract state s. Each episode of this subtask consists of d(s, s) × H worker timesteps (longer transitions need more timesteps to traverse), where the reward at each timestep is R (s,s) (x t) = 1 if the skill has successfully reached the end of the transition (φ(x t) = s ) and 0 otherwise. These episodes additionally terminate if the main episode terminates or if the manager receives negative environment reward. When solving these subtasks to construct the action set of the abstract MDP, the worker must be careful not to violate the Markov property. In particular, the concrete state may contain some historydependent information lost due to the state abstraction function. For example, consider the task of jumping over a dangerous hole, consisting of three abstract states: s 1 (the cliff before the hole), s 2 (the air above the hole), and s 3 (the solid ground on the other side of the hole). The worker might incorrectly assume that it can reliably traverse from s 1 to s 2 by simply walking off the cliff. But adding this as a reliable transition to the abstract MDP causes a problem: there is now no way to successfully traverse from s 2 to s 3 due to missing history-dependent information in the abstract state (i.e., the way the worker navigated s 1 to s 2), violating the Markov property. On navigating a transition (s, s), the worker avoids this problem by navigating to s and then checking for history-dependent consequences with the holding heuristic. The worker assumes that if its navigation of (s, s) changed unobserved parts of the state, then those changes would eventually cause the abstract state to change (e.g., in the example, the worker would eventually hit the bottom and die). Consequently, if the worker can stay in s for many timesteps, then it did not significantly unobserved parts of the state. This corresponds to only declaring the episode as a success if the worker accumulates at least R hold reward (equivalent to being in s for R hold timesteps).Any RL algorithm can be used to represent and train the skills to perform this subtask. We choose to represent each skill as a Dueling DDQN (van ; BID47 . For faster training, the skills use self-imitation BID29 to more quickly learn from previous successful episodes, and count-based exploration similar to BID5 to more quickly initially discover skill reward. Since the skill inventory can contain many skills, we save parameters by occasionally using pixel-blind skills. Appendix A.3 fully describes our skill training and architecture. We are interested in the sample complexity BID16, the number of samples required to learn a policy that achieves reward close to the optimal policy, with high probability. Standard in the tabular setting (e.g., MBIE-EB BID38) guarantee learning a nearoptimal policy, but require a number of timesteps polynomial in the size of the state space, which is effectively vacuous in the deep RL setting, where state spaces can be exponentially large (e.g., > 10 100 states). In contrast, our approach is able to use time and space polynomial in the size of the abstract state space, which is exponentially smaller, by operating on the abstract MDP.Formally, assuming that our neural network policy class is rich enough to represent all necessary skills, with high probability, our approach can learn a near-optimal policy on a subclass of MDPs in time and space polynomial in the size of the abstract MDP (details in Appendix C). The key intuition is that instead of learning a single task where the time horizon is the length of the game, our approach learns many subtasks where the time horizon is the number of steps required to navigate from one abstract state to another. This is critical, as many deep RL algorithms (e.g. -greedy) require a number of samples exponential in the time horizon to solve a task. Following BID2, we empirically evaluate our approach on three of the most challenging games from the ALE BID6: MONTEZUMA'S REVENGE, PITFALL!, and PRIVATE EYE. We do not evaluate on simpler games (e.g., Breakout), because they are already solved by prior state-of-the-art methods BID13 and do not require sophisticated exploration. We use the standard ALE setup (Appendix A) and end the episode when the agent loses a life. We report rewards from periodic evaluations every 4000 episodes, where the manager plans for optimal reward in the currently constructed abstract MDP. We average our approach over 4 seeds and report 1 standard deviation error bars in the training curves. Our experiments use the same set of hyperparameters (Appendix A.1) across all three games, where the hyperparameters were exclusively and minimally tuned on MONTEZUMA'S REVENGE.In all three games, the state abstraction function uses the RAM state, available through the ALE simulator, to extract the bucketed location of the agent and the agent's current inventory. Roughly, this distinguishes states where the agent is in different locations or has picked up different items, but doesn't distinguish states where other details differ (e.g. monster positions or obstacle configurations). Notably, the abstract state function does not specify what each part of the abstract state means, and the agent does not know the entire abstract state space beforehand. We describe the exact abstract states in Appendix A.2. Among the many deep RL approaches, in each game, we compare with the prior non-demonstration state-of-the-art approach, which use prior knowledge comparable to our RAM information:• In MONTEZUMA'S REVENGE, we compare with SmartHash BID44, a countbased exploration approach which estimates state visit counts with a hash-based density model and provides intrinsic reward to revisit states with low visit counts. Like our approach, SmartHash also uses RAM state information. It hashes each state to a hand-selected subset of the RAM state and maintains visit counts on the hashes. • In PITFALL!, we compare with SOORL BID18, a planning approach which requires prior knowledge to extract the objects on the screen. SOORL is the only prior nondemonstration approach to achieve positive reward in PITFALL!, but requires extensive engineering (much stronger than RAM state info) to identify and extract all objects. Once SOORL has access to the objects, it learns in very few frames since data from similar objects can be pooled in learning. Consequently, in our training curves, we report its final average performance over 100 runs, as well as its final best performance over 100 runs, • In PRIVATE EYE, we compare with another count-based exploration method, DQNPixelCNN, which uses a pixel-based density model to estimate state visitation counts. We compare with the reported in. DQNPixelCNN uses less prior knowledge than our approach, but we compare with it because it achieves the previous non-demonstration state-of-the-art .• In all three games, we compare with AbstractStateHash, which performs count-based exploration identical to SmartHash, but uses the same RAM information as our approach. Comparison between our worst performing seed with the best performing seed of the prior-state-of-the-art. Our worst performing seed outperforms the prior best on MONTEZUMA'S REVENGE and PITFALL! and performs comparably to the prior best on PRIVATE EYE. Our best performing seed achieves new peak rewards. FIG0 shows the main . AbstractStateHash matches the prior state-of-the-art on MON-TEZUMA'S REVENGE, but performs relatively poorly on PRIVATE EYE and PITFALL!. This suggests both that prior state-of-the-art methods do not effectively leverage the state abstraction function, and that the state abstraction function does not trivialize the learning problem. In MONTEZUMA'S REVENGE, after 2B training frames, our approach achieves a final average reward of 11020, more than doubling the average reward of SmartHash: 5001. Our approach achieves higher average reward than SmartHash at every point along the training curves and continues to learn even late into training, while SmartHash plateaus (Appendix B.3 presents more on the ability of our approach to continue to learn without plateauing).Our approach is the first non-demonstration approach to achieve superhuman performance on PIT-FALL!, achieving a final average reward of 9959.6 after 2B frames of training, compared to average human performance: 6464 BID33. In addition, our approach achieves more than double the reward of SOORL, which achieves a maximum reward of 4000 over 100 seeds and a mean reward of 80.52, and even significantly outperforms Ape-X DQfD BID33, which uses high-scoring expert demonstrations during training to achieve a final mean reward of 3997.5.In PRIVATE EYE, our approach achieves a mean reward of 35636.1, more than double the reward of DQN-PixelCNN, which achieves 15806.5. Our approach performs even better, approaching human performance, if we change a single hyperparameter (Appendix B.4).Stability. Recent work BID12 has drawn attention to the instability of deep RL . To highlight the stability of our , we compare our worst performing seed against the prior state-of-the-art's best performing seed in TAB3. Even our worst seed outperforms the mean performance of the prior state-of-the-art approaches. In addition, our worst seed is competitive with the highest previously reported rewards in each of the games, significantly outperforming the previous high in MONTEZUMA'S REVENGE and PITFALL!, and narrowly performing worse than the previous high in PRIVATE EYE. Even in PRIVATE EYE, while DQN-PixelCNN achieves 39000 reward on its best single episode across all seeds, none of its seed consistently achieves more than 15806.5 reward over many episodes. In contrast, our worst seed consistently obtains 36200 reward. Furthermore, our best seeds achieve new peak performances in each of the games. By using the abstract MDP, our approach can quickly generalize to new tasks in the same environment that were not seen during training. It does this by revisiting the transitions in the abstract MDP to update the rewards model with the newly observed reward. We study the ability of our approach to do this. After our approach completes training on the original reward function on MONTEZUMA'S REVENGE, we evaluate its performance on three new reward functions, allowing our approach to interact with the environment for an additional 1M frames to observe each new reward function. We compare with SmartHash, trained from scratch directly on the new reward functions for 1B frames. Even when evaluated on an unseen reward function, our approach achieves about 3x as much reward as SmartHash, which is directly trained on the new reward function. This suggests that our approach can generalize to new tasks with the same dynamics. We describe the details in Appendix B.2. We additionally evaluate the performance of our approach on the recommended BID23 form of ALE stochasticity (sticky actions 25% of the time) on PRIVATE EYE (selected because it requires the fewest frames for training). Figure 3 compares the performance of our method on the stochastic version of PRIVATE EYE with the performance of our method on the deterministic version of PRIVATE EYE. Performance degrades slightly on the stochastic version, because the worker's skills become harder to learn. However, both versions outperform the prior state-of-the-art DQNPixelCNN, and the worker is able to successfully abstract away stochasticity from the manager in the stochastic version of the game so that the abstract MDP remains near-deterministic. To see how easy it is to create a high-performing state abstraction function on new tasks, we study the robustness of our approach to state abstraction functions of varying degrees of coarseness on PRIVATE EYE. Our state abstraction function buckets the agent's (x, y) coordinates. We vary the coarseness of the abstraction function by varying the bucketing size: increasing the bucketing size in fewer, coarser abstract states. We report in Figure 4 on five different bucket sizes obtained by scaling the original bucket size by 1 2, 2 3, 1, 3 2, and 2. To adjust for the updated bucket sizes, we also scale the worker's skill episode horizon H worker by the same value. Our method outperforms the prior state-of-the-art approach DQN-PixelCNN across the entire range of bucket sizes, suggesting that our approach does not require a highly tuned state abstraction function. Exploration in tabular settings is well-understood via optimism in the face of uncertainty (OFU) BID7 BID39 ) and posterior sampling BID31 BID30. OFU methods BID7 BID37 ) achieve provably near-optimal policies in polynomial time by providing reward bonuses for exploring regions of uncertainty. Nevertheless, despite recent stronger optimality bounds BID8, these methods do not scale to the deep RL setting, where the state space is prohibitively large. BID5; BID44; similarly apply optimism reward bonuses in the case of high-dimensional state spaces, empirically improving exploration. However, these methods no longer guarantee optimality, and can suffer from insufficient exploration because they reactively seek infrequently visited states, whereas our manager proactively seeks new abstract states. While model-based RL succeeds in the tabular setting BID39 ) and in tasks with relatively small (e.g., < 100 dimensions) state spaces BID25, it has little success on tasks with exponentially large state spaces (e.g., pixel inputs). Prior work BID27 that learns models in these large state spaces suffers from compounding errors BID42, where longterm predictions become extremely inaccurate. To make matters worse, while prior model-based works BID49 BID11 ) succeed on relatively dense reward tasks, model-based planning methods (e.g., value iteration) can be computationally intractable in longhorizon, sparse-reward tasks, even when a perfect model is known. To circumvent these problems, our work learns an abstract MDP consisting of an exponentially smaller abstract state space and learned skills. BID28 similarly learns a model over abstract states and skills, but uses manually engineered skills, whereas ours are learned. Our work relates to prior work on hierarchical reinforcement learning (HRL), which also operates on abstract states BID10 BID22 with learned skills or subgoals BID35 BID36 BID46 BID4. However, a key difference is that our work constructs the abstract MDP, enabling us to perform targeted exploration via planning and rely on the Markov property to avoid exponentially large state histories. In contrast, the abstract states and skills in these other works do not meet such useful structural properties, and consequently can be difficult to learn with. BID34 similarly constructs an abstract MDP like ours. However, due to critical design decisions, our approach outperforms theirs by nearly an order of magnitude. Whereas our approach monotonically grows the known set by saving worker parameters as transitions become reliable, BID34 uses the same worker parameters to simultaneously learn many transitions. This causes catastrophic forgetting, as training on a new transition causes the worker to fail a previously learned transition, and prevents growth of the abstract MDP.Only imitation learning methods BID2 BID33 outperform our method on the ALE's hardest exploration games. However, using demonstrations sidesteps the exploration problem our approach seeks to solve because following demonstrations leads to high reward. This work presents a framework for tackling long-horizon, sparse-reward, high-dimensional tasks by using abstraction to decrease the dimensionality of the state space and to address compounding model errors. Empirically, this framework performs well in hard exploration tasks, and theoretically guarantees near-optimality. However, this work has limitations as well. First, our approach relies on some prior knowledge in the state abstraction function, although we compare against state-ofthe-art methods using a similar amount of prior knowledge in our experiments. This information is readily available in the ALE, which exposes the RAM, and in many robotics tasks, which expose the underlying state (e.g., joint angles and object positions). Still, future work could attempt to automatically learn the state abstraction or extract the abstraction directly from the visible pixels. One potential method might be to start with a coarse represention, and iteratively refine the representation by splitting abstract states whenever reward is discovered. Another limitation of our work is that our simple theoretical guarantees require relatively strong assumptions. Fortunately, even when these assumptions are not satisfied, our approach can still perform well, as in our experiments. A , the pixel concrete states are downsampled and cropped to 84 by 84 and then are converted to grayscale. To capture velocity information, the worker receives as input the past four frames stacked together. Every action is repeated 4 times. In addition, MONTEZUMA'S REVENGE and PITFALL! are deterministic by default. As a , the manager deterministically navigates to the fringes of the known set by calling on the worker's deterministic, saved skills. To minimize wallclock training time, we save the states at the fringes of the known set and enable the worker to teleport to those states, instead of repeatedly re-simulating the entire trajectory. When the worker teleports, we count all the frames it would have had to simulate as part of the training frames. Importantly, this only affects wallclock time, and does not benefit or change the agent in any way. Notably, this does not apply to PRIVATE EYE, where the initial state is stochastically chosen from two similar possible states. A.1 HYPERPARAMETERS All of our hyperparameters are only tuned on MONTEZUMA'S REVENGE. Our skills are trained with the Adam optimizer BID19 with the default hyperparameters. TAB5 describes all hyperparameters and the values used during experiments (bolded), as well as other values that we tuned over (non-bolded). Most of our hyperparameters were selected once and never tuned. In MONTEZUMA'S REVENGE, each abstract state is a (bucketed agent x-coordinate, bucketed agent y-coordinate, agent room number, agent inventory, current room objects, agent inventory history) tuple. These are given by the RAM state at indices 42 (bucketed by 20), 43 (bucketed by 20), 3, 65, and 66 respectively. The agent inventory history is a counter of the number of times the current room objects change (the room objects change when the agent picks up an object).In PITFALL!, each abstract state is a (bucketed agent x-coordinate, bucketed agent y-coordinate, agent room number, items that the agent has picked up) tuple. These are given by the RAM state at indices 97 (bucketed by 20), 105 (bucketed by 20), 1, and 113 respectively. In PRIVATE EYE, each abstract state is a (bucketed agent x-coordinate, bucketed agent y-coordinate, agent room number, agent inventory, agent inventory history, tasks completed by the agent) tuple. These are given by the RAM state at indices 63 (bucketed by 40), 86 (bucketed by 20), 92, 60, 72, and 93 respectively. Architecture. Our skills are represented as Dueling DDQNs BID45 BID47, which produce the state-action value Q (s,s) (x, a) = A (s,s) (x, a) + V (s,s) (x), where A (s,s) (x, a) is the advantage and V (s,s) (x) is the state-value function. The skills recover a policy π K(s,s) (a|x, (s, s)) by greedily selecting the action with the highest Q-value at each concrete state x. The skill uses the standard architecture BID24 to represent A (s,s) (x, a) and V (s,s) (x) with a small modification to also condition on the transition (s, s). First, after applying the standard ALE pre-processing, the skill computes the pixel embedding e x of the pixel state x by applying three square convolutional layers with (filters, size, stride) equal to,, and respectively with rectifier non-linearities BID26, and applying a final rectified linear layer with output size 512. Next, the skill computes the transition embedding e (s,s) by concatenating [e r ; e dif f] and applying a final rectified linear layer with output size 64, where:• e r is computed as the cumulative reward received by the skill during the skill episode, represented as one-hot, and passed through a single rectified linear layer of output size 32.• e dif f is computed as s − s passed through a single rectified linear layer of output size 96.Finally, e x and e (s,s) are concatenated and passed through a final linear layer to obtain A (s,s) (x, a) and V (s,s) (x).To prevent the skill from changing rapidly as it begins to converge on the optimal policy, we keep a sliding window estimate of its success rate p success. At each timestep, with probability 1 − p success, we sample a batch of (x, a, r, x) tuples for transition (s, s) from the replay buffer and update the policy according the DDQN loss function: L = ||Q (s,s) (x, a) − target|| 2 2, where target = (r +Q target (x, arg max a ∈A Q (s,s) (x, a))). Additionally, since the rewards are intrinsically given, the optimal Q-value is known to be between 0 and R hold. We increase stability by clipping target between these values. Pixel blindness. In addition, some skills are easy to learn (e.g. move a few steps to the left) and don't require pixel inputs to learn at all. To prevent the skills from unnecessarily using millions of parameters for these easy skills, the worker first attempts to learn pixel-blind skills for simple transitions (s, s) with d(s, s) = 1 (i.e. (s, s) was directly observed by the manager). The pixelblind skills only compute e (s,s) and pass this through a final layer to compute the advantage and value functions (they do not compute or concatenate with e x). If the worker fails to learn a pixelblind skill, (e.g. if the skill actually requires pixel inputs, such as jumping over a monster) it will later try to learn a pixel-aware skill instead. Epsilon schedule. The skills use epsilon-greedy exploration, where at each timestep, with probability, a random action is selected instead of the one produced by the skill's policy BID48. Once a skill becomes frozen, is permanently set to 0.The number of episodes required to learn each skill is not known in advance, since some skills require many episodes to learn (e.g. traversing a difficult obstacle), while other skills learn in few episodes (e.g. moving a little to the left). Because of this, using an epsilon schedule that decays over a fixed number of episodes, which is typical for many RL algorithms, is insufficient. If epsilon is decayed over too many episodes, the simple skills waste valuable training time making exploratory actions, even though they've already learned near-optimal behavior. In contrast, if epsilon is decayed over too few episodes, the most difficult skills may never observe reward, and may consequently fail to learn. To address this, we draw motivation from the doubling trick in online learning BID1 to create an epsilon schedule, which accomodates skills requiring varying number of episodes to learn. Instead of choosing a fixed horizon, we decay epsilon over horizons of exponentially Figure 5: The saw-tooth epsilon schedule used by our skills increasing length, summarized in Figure 5. This enables skills that learn quickly to achieve low values of epsilon early on in training, while skills that learn slowly will later explore with high values of epsilon over many episodes. Count-based exploration. Our skill additionally use count-based exploration similar to BID44; BID5 to learn more quickly. Each skill maintains a count of the number of times visit(s) it has visited each abstract state s. Then, the skill provides itself with additional intrinsic reward to motivate itself to visit novel states, equal to. Self-imitation. When learning to traverse difficult obstacles (e.g. jumping over a disappearing floor), the skill may observe a few successes long before successfully learning a policy to reliably traverse the difficult obstacle. We use a variant of the self-imitation described in BID29 to decrease this time. Whenever a skill successfully traverses a transition, it adds the entire successful trajectory to a separate replay buffer and performs imitation learning on the successful trajectories. These successful trajectories are actually optimal skill trajectories because the skill episode uses undiscounted reward, so all successful trajectories are equally optimal. To update on these skills, the skill periodically samples from this replay buffer and updates on an imitation loss function DISPLAYFORM0, where θ is the skill's parameters, and L 1 and L 2 are defined as below:• Let G t = T i=t r t be the reward to-go for a successful trajectory a) on the reward to-go of the successful trajectory, because G t is actually the optimal Q-value on successful trajectories (all successful trajectories are equally optimal): i.e., L 1 = ||G t − Q (s,s) (x t, a t)|| 2. DISPLAYFORM1 • We use the margin-loss from DISPLAYFORM2 Intuitively, L 2 encourages the skill to replay the actions that led to successful trajectories over other actions. We use λ = 0.5, which was chosen with no hyperparameter tuning. B ADDITIONAL The worker learns skills that successfully apply in to many similar transitions. Figure 6 depicts the number of different transitions each skill is used on in MONTEZUMA'S REVENGE, PITFALL!, and PRIVATE EYE. The simplest skills (e.g. move to the left) enjoy the highest number of reuses, while more esoteric skills (e.g. jump over a particular monster) are only useful in few scenarios. Figure 7 provides an example of a skill in MONTEZUMA'S REVENGE with relatively high reuse. The arrows denote the movement of the agent when it executes the skill. The same skill that jumps over a monster in the first room (Figure 7(a) ) can also climb up ladders. In Figure 7(b), the skill appears to know how to climb up all parts of the ladder except for this middle. This occurs because the spider occasionally blocks the middle of the ladder, and a different special skill must be used to avoid the spider. However, the skill reuse is not perfect. For example, in Figure 7 (a), the skill can climb up the top half of ladders, but a separate skill climbs the bottom half of the ladders. To evaluate the ability of our approach to generalize to new reward functions, we train our approach on the basic MONTEZUMA'S REVENGE reward function and then test it on three challenging new reward functions (illustrated in FIG4), not seen during training:• Get key: the agent receives 1000 reward for picking up the key in room 14 (6 rooms away from the start). In addition, the agent receives -100 reward for picking up any other objects or opening any other doors. Figure 9: Training curves of SmarthHash on alternate tasks in MONTEZUMA'S REVENGE, compared with the performance of our approach generalizing to the new task. Our approach only receives 1M frames with the new task.• Kill spider: the agent receives 1000 reward for killing the spider in room 13 (5 rooms away from the start). To kill the spider, the agent must first pick up the sword in room 6 (3 rooms away from the start) and save the sword for the spider. The agent receives no other reward.• Enter room 8: the agent receives 1000 reward for entering room 8 (6 rooms away from the start). The agent receives no other reward. In all three tasks, the episode ends when the agent completes its goal and receives positive reward. Our approach trains on the basic MONTEZUMA'S REVENGE reward function for 2B frames, and then is allowed to observe the new reward functions for only 1M frames. We compare with SmartHash, which trains directly on the new reward functions for 1B frames. The are summarized in Figure 9. Even when evaluated on a reward function different from the reward function it was trained on, our approach achieves about 3x as much reward as SmartHash, which is trained directly on the new reward function. Averaged over all 3 tasks, our approach achieves an average reward of 716.7 out of an optimal reward of 1000, whereas SmartHash only achieves an average reward of 220, even when trained directly on the new reward function. These experiments suggest that after our approach is trained in an environment on one task, it can quickly and successfully adapt to new tasks with little additional training. Whereas many prior state-of-the-art approaches tend to plateau toward the end of training, our approach continues to make near-linear progress. Figure 10 graphs the number of transitions learned by the worker against the number of frames in training. In MONTEZUMA'S REVENGE and PIT-FALL! particularly, the rate the worker learns new transitions is nearly constant throughout training. Because of this, when we continued to train a single seed on PITFALL!, by 5B frames, it achieved a reward of 26000, and by 20B frames, it achieved a reward of 35000. By changing a single hyperparameter, our approach can perform even better on PRIVATE EYE, exceeding human performance on 2 of 4 seeds. Since nearby abstract states are particularly easy to discover in PRIVATE EYE, the manager needs not explore for new transitions as many times. Consequently, if we decrease the number of times the manager explores around each abstract state N visit from the 500 (used in the main experiments for all three games) to 10, performance improves. Figure 11 compares the performance with the decreased value of N visit with the original value of N visit reported in the main experiments. Decreasing N visit prevents the manager from wasting frames with unnecessary exploration and consequently enables the worker to learn more transitions in fewer total frames. With N visit set to 10, our approach achieves a final average performance of 60247 after 200M frames of training. Additionally, the top 2 of our 4 seeds achieve rewards of 75600 and 75400, exceeding average human performance: 69571 BID33. In general, a hierarchical policy over skills is not guaranteed to be near-optimal, because certain optimal trajectories may be impossible to follow using the skills. Because of this, hierarchical reinforcement learning literature typically focuses on hierarchical optimality BID10 optimality given the abstractions. However, under the following assumptions, our approach provably achieves a near-optimal policy on the original MDP in time polynomial in the size of the abstract MDP, with high probability. Proposition 1. Under the assumptions, for a given input η and, π t is at most suboptimal, To prove Proposition 1, we require the following three lemmas: DISPLAYFORM0 Lemma 1. By setting N transition to be O(log 1−(1−η) 1 |Φ| 2 /2 2(/|Φ|HV * (x0)) 2 ), with probability 1 − η, at each timestep t, π t is near-optimal on the current abstract MDP: i.e., V πt t (s) ≥ V * t (s)− for all abstract states s ∈ Φ.Lemma 2. If the known set is equal to the set of all abstract states (S = Φ) at timestep T, then for any policy π on the abstract MDP, π achieves the same expected reward on the abstract MDP as on the concrete MDP: i.e., V π T (φ(x 0)) = V π (x 0), where x 0 is the initial concrete state. In addition, the expected return of the optimal policy on the abstract MDP is equal to the expected return of the optimal policy on the concrete MDP: i.e., V * T (φ(x 0)) = V * (x 0) where x 0 is the initial state. Lemma 3. With probability 1 − η, the known set grows to cover all abstract states in O |Φ| 3 (|A| + Given these lemmas, we are ready to prove Proposition 1:Proof of Proposition 1. For simplicity, we ignore terms due to appropriately setting N transition and 1 − η from Lemma 1, but these terms are all polynomial in the size of the abstract MDP.By Lemma 3 the known set grows to cover all abstract states in T = O |Φ| 3 (|A| + log K|Φ|+log 1 η log 1 p) + d max × H worker timesteps. For all timesteps t ≥ T, by Lemma 1, π t is at most suboptimal on the abstract MDP. On all those timesteps, the known set is equal to all abstract states, so by Lemma 2, π t is at most suboptimal on the concrete MDP.Proofs. Now, we prove Lemma 1, Lemma 2, and Lemma 3.Proof of Lemma 1. LetP (s |o, s) denote the estimated transition dynamics 3 andR(s, s) denote the estimated reward model in the abstract MDP.For each reliable transition (s, s) (action in the abstract MDP), the manager estimatesP (s |o, s) from N transition samples of the worker. We bound the error in the model |P (s |o, s) − P (s |o, s)| with high probability by Hoeffding's inequality: DISPLAYFORM1 By the Assumption 2,R(s, s) = R(s, s) because all trajectories leading to s achieve some reward r and all trajectories leading to s achieve some reward r, so a single estimate R(s, s) = r − r is sufficient to accurately determineR.Because the model errors are bounded, and because the abstract MDP is Markov, we can apply the simulation lemma BID17, which states that if |P (s |o, s) − P (s |o, s)| ≤ α and |R(s, s)−R(s, s)| ≤ α, then the policy π optimizing the MDP formed byP andR is at most suboptimal: i.e., at each timestep t, V π t (s) ≥ V * t (s)− for all s ∈ S, where α is O (/|S|HV * (x 0)) 2, and H is the horizon of the abstract MDP. Since the total number of transitions is bounded by |S| 2, substituting for N transition gives the desired .

We automatically construct and explore a small abstract Markov Decision Process, enabling us to achieve state-of-the-art results on Montezuma's Revenge, Pitfall!, and Private Eye by a significant margin.

Represent the TLDR for finding the corresponding scientific article

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: represent the following document

Question: Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. This is a critical shortcoming for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms which use KL-control to penalize divergence from a pre-trained prior model of probable actions. This KL-constraint reduces extrapolation error, enabling effective offline learning, without exploration, from a fixed batch of data. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. This Way Off-Policy (WOP) algorithm is tested on both traditional RL tasks from OpenAI Gym, and on the problem of open-domain dialog generation; a challenging reinforcement learning problem with a 20,000 dimensional action space. WOP allows for the extraction of multiple different reward functions post-hoc from collected human interaction data, and can learn effectively from all of these. We test real-world generalization by deploying dialog models live to converse with humans in an open-domain setting, and demonstrate that WOP achieves significant improvements over state-of-the-art prior methods in batch deep RL. In order to scale deep reinforcement learning (RL) to safety-critical, real-world domains, two abilities are needed. First, since collecting real-world interaction data can be expensive and timeconsuming, algorithms must be able to learn from off-policy data no matter how it was generated, or how little correlation between the data distribution and the current policy. Second, it is often necessary to carefully test a policy before deploying it to the real world; for example, to ensure its behavior is safe and appropriate for humans. Thus, the algorithm must be able to learn offline first, from a static batch of data, without the ability to explore. This off-policy, batch reinforcement learning (BRL) setting represents a challenging RL problem. Most deep RL algorithms fail to learn from data that is not heavily correlated with the current policy (b). Even models based on off-policy algorithms like Q-learning fail to learn in the offline, batch setting, when the model is not able to explore. If the batch data is not sufficient to cover the state-action space, BRL models can suffer from extrapolation error, learning unrealistic value estimates of state-action pairs not contained in the batch (b). It can be impossible to correct for extrapolation error when there is a mismatch in the distribution of stateactions pairs in the batch data, and the distribution induced by the learned policy. For example, if the policy learns to select actions which are not contained in the batch, it cannot learn a reasonable value function for those actions. Figure 1 illustrates this concept, where the batch only covers a subset of possible policies. Extrapolation error is particularly problematic in high-dimensional state and action spaces (such as those inherent in language generation). We propose to resolve these issues by leveraging a pre-trained generative model of the state-action space, p(a|s), trained on known sequences of interaction data. While training with RL, we penalize divergence from this prior model with different forms of KL-control. This technique ensures that the RL model learns a policy that stays close the state-action distribution of the batch, combating Figure 1: In this example batch RL problem, the robot's goal is to travel the minimum distance around the black walls to get to the red flag. A trained behavior policy generated the batch data; the probability of each of the states appearing in the batch, p B (s), is in yellow (white locations are not contained in the batch). If the offline RL policy estimates the value of going up or left from the start position is high, it will have no way to refine this estimate using the batch data, or learn a good policy in this region of state space. The KL-constraint ensures that the RL policy will stay within the support of the batch data. However, the behavior policy is suboptimal, so using behavior cloning to directly imitate the batch data will in suboptimal return. Instead, the KL-constrained model can learn to find the optimal policy, which is within the support of the batch. extrapolation error. We also propose using dropout to obtain uncertainty estimates of the target Qvalues, and use this lower bound to alleviate overestimation bias. We benchmark against a discrete adaptation of Batch Constrained Q-learning (BCQ) (b), a recently proposed state-of-the-art BRL algorithm for continuous domains, and show that our Way Off-Policy algorithm achieves superior performance in both a traditional RL domain, as well as in a challenging, underexplored, real-world reinforcement learning problem: using implicitly expressed human reactions in chat to improve open-domain dialog systems. When a machine learning system interacts with humans, ideally we would like to learn about the humans' preferences in order to improve its performance. Yet having humans manually indicate their preferences through explicit means like pressing a button (e.g.) or submitting a feedback report, does not scale. Instead, we would like to be able to use humans' implicit reactions, such as the sentiment they express, or the length of the conversation, in order to improve the policy. However, applying off-policy batch RL to language generation is challenging because the number of potential combinations of words and sentences leads to a combinatorial explosion in the size of the state space. The action space -the set of frequent vocabulary words in the English language -is 20,000-dimensional. This compounds extrapolation error, making BRL even more difficult. However, when learning from human interactions in the wild, it is crucial to be able to learn offline and test the policy before deploying it, lest it learn inappropriate behaviors (e.g.). To support this work, we developed an interactive online platform that allows humans to chat with deep neural network dialog models running on a GPU; the BRL models trained for this study are available live at https://neural.chat/rl/. Through this platform we collected human responses to a set of over 40 different dialog models over the course of several months. Using our Way Off-Policy algorithm, we are able to effectively learn from this batch of data, in spite of the fact that it was generated with a vastly different set of model architectures, which were trained on different datasets. Further, we use the batch to learn from many different reward functions designed post-hoc to extract implicit human preferences, something that is only possible with effective off-policy BRL. In summary, the contributions of this paper are: • A novel algorithm, Way Off-Policy learning, which is the first to propose using KL-control from a pre-trained prior model as a way to reduce extrapolation error in batch RL. • Experiments showing the effectiveness of WOP above strong baselines based on prior work (e.g. Fujimoto et al. (2018b) ), on both traditional RL tasks and on the challenging problem of open-domain dialog generation. • A set of novel conversation rewards based on how human preferences are implicitly expressed in text. We are the first work to learn from implicit signals in conversation offline using batch RL. The approach we propose is based on KL-control, a branch of stochastic optimal control (SOC) where the Kullback-Leibler (KL) divergence from some distribution is used to regularize an RL policy (e.g. (; ; ;) ). Well-known examples include Trust Region Policy Optimization (TRPO) , and use conservative, KL-regularized policy updates to restrict the RL algorithm to stay close to its own prior policy (e.g. (; ; ;) ). KL-control can also be applied to entropy maximization (e.g. (; ;) ); for example, G-learning penalizes KLdivergence from a simple uniform distribution in order to cope with overestimation of Q-values .KL-control has also been used to improve transfer learning between maximum likelihood estimation (MLE) training on data, and training with RL . To the best of our knowledge, our work is the first to propose penalizing KL-divergence from a learned prior model of the state-action space as a way to improve offline batch RL. Other strategies to improve off-policy learning have been proposed, but differ from this work in key respects. Many focus on scenarios where the policy is able to explore and collect more data (e.g. ;); for example, when learning online from an outdated replay buffer (e.g.). In contrast, we learn entirely offline, from a fixed batch of data, without the ability to explore. Methods proposed for this setting have often not been used in conjunction with modern function approximation techniques (e.g.). Many other works focus on off-policy policy evaluation (rather than policy learning), for example using importance sampling or model estimation (e.g. ; ; ;). In the deep BRL setting, have proposed a correction to policy gradients, have proposed covariance-shift methods, and have proposed normalized feature representations. use maximum mean discrepancy to cope with extrapolation error in BRL, while use a Random Ensemble Mixture (REM) Q-network. Most similar to our work is Batch Constrained Q-learning (BCQ) (b), which tackles off-policy deep BRL in continuous action domains by training a generative model of the batch, p(a|s), sampling from this model, and selecting the best action based on a Q-estimate. Unlike our approach, this does not integrate information about the distribution p(a|s) directly into the policy, or allow the model to learn when to strategically deviate from the prior in order to obtain more reward. We propose using dropout to approximate model uncertainty of the target Q-network. The idea of using dropout to estimate uncertainty in neural networks was proposed by. Different forms of uncertainty estimates have been used in RL (e.g. ;); for example, Bayesian uncertainty estimates have been proposed as an alternative to double DQN . Improving dialog systems with RL has largely been restricted to task-oriented dialog systems, which have a limited number of task-specific actions (e.g.). These approaches may incorporate human input, usually through explicit, manual feedback (e.g.), but sometimes with more implicit signals, such as the user interrupting the system or starting over . Efforts to expand RL to the open-domain dialog setting, such as those of Li et al. (2016b;, are less numerous, and do not involve learning from human feedback. Even in the open-domain setting, authors may choose to use a highly restricted action space; for example, using RL to choose which scripted or MLE dialog model to invoke to answer a user's query (a). Since the posting of the preprint of this paper, have used explicit human feedback to improve the summarization and text continuation performance of a large-scale language model. Although they do not study dialog or the batch RL setting (instead learning online from a trained model of human feedback), they do make use of our proposal to penalize KL-divergence from a pre-trained language model, and find that this is important to achieving good performance. Although implicit signals such as sentiment and conversation length have been used in MLE systems, the idea of using such signals as a reward for RL is relatively unexplored. Shin and colleagues uses on-policy learning in conjunction with a usersentiment approximator to improve a seq2seq model , but are unable to learn directly from user feedback. To the best of our knowledge, we are the first to use batch RL to train open-domain dialog models on implicit cues gained from real human interactions. We employ typical RL notation in which s t represents the environment state at time t, the agent takes action a t according to its policy π(a t |s t), and receives reward r(s t, a t). The agent's goal is to maximize reward over an episode trajectory τ, with a discount factor of γ applied to future rewards. Q-learning learns an action-value estimate of the total expected discounted future reward,, through iterative updates based on the Bellman equation: In deep Q-learning , a Q-network approximates Q θπ (s t, a t) and drives the policy π. A second target Q-network approximates the expected reward from the next state, ). In batch RL, we are given a fixed batch of data B, and assume that no further interaction with the environment is possible. To train Q θπ, we sample (s t, a t, r t, s t+1) ∼ B, and update the weights of the Q-network to approximate Eq. 1. Because Q-learning is an off-policy algorithm, in principle it should be able to learn from data collected by any behavior policy. However, extrapolation error can occur if the BRL policy learns to favour a state-action pair (s, a) that is unlikely, or not contained, in the batch data. In this case, the estimate Q(s, π(s)) can be arbitrarily bad (b). Such errors can then accumulate through the Bellman backup operator . Experiments from Fujimoto et al. (2018b) show that extrapolation error can be highly detrimental to learning off-policy in BRL. These problems are compounded by the fact that algorithms based on the Bellman operator are inherently optimistic in the face of uncertainty. When value estimates for some region of the stateaction space are noisy (because too few experience samples have been used to refine them), the maximum operation in Eq. 1 will lead to an overestimation of expected future reward. In a normal RL setting, this overestimation bias drives the model to explore areas of the state-action space for which the value estimates have the highest variance, thus enabling it to refine them; in essence, creating a built-in drive to explore. However, in a batch setting where exploration is not possible, the model is instead driven to value parts of the state-action space for which it has little to no data to learn a good policy (see Figure 1). The overestimation of Q-values in the BRL setting necessitates other methods for estimating the future reward via the Target Q-network. Clipped Double Q-learning (a) maintains two independent pairs of Q-networks, and takes the minimum of their estimates of future reward. This approach is computationally expensive and memory intensive. Instead, we leverage the fact that a network trained with dropout can be used to approximate a Bayesian uncertainty estimate of the output value . Given the target Q-network Q θ T, we compute Q(a t+1, s t+1) using a Monte Carlo (MC) estimate of the lower-bound of Q θ T (a t+1, s t+1) by running M stochastic forward passes of the network, each with a new dropout mask Using the minimum operator penalizes high variance estimates and leads the algorithm to be pessimistic in the face of uncertainty, rather than optimistic. Such a bias will push the model to favour actions that lead to states well covered by the batch data. Batch Constrained Q-learning (BCQ) (b) proposes to address the BRL problem by constraining the actions of the Q-network to be close to the data contained within the batch. This is accomplished by learning a generative model of the batch, G w = p(a|s), and sampling from this model during learning and inference. Because BCQ is designed for continuous action domains, it applies a learned perturbation model ξ(s, a; Φ) which is allowed to alter the action within the range [−Φ, Φ]. BCQ learns Q-estimates that incorporate the perturbation model, Q θ (s, a + ξ(s, a; Φ)). To act, n possible actions are sampled from the generative model,, perturbed, and the action with the maximum Q-value is selected, giving the BCQ policy: We propose an adaptation of BCQ to discrete action spaces (DBCQ) which does not use a continuous perturbation model. Since BCQ relies on Double Clipped Q-learning (a), here we use dropout-based uncertainty estimates as in Eq. 2. Thus the DBCQ policy is: Rather than simply sample from the prior, we would like the Q-learning algorithm to directly incorporate the prior into the policy. Thus, we use KL-control to penalize divergence between the learned prior p(a|s), and the Q-network policy π θ, while still maximizing reward. Given a trajectory of ac- be the policy of our Q-learning algorithm at the trajectory level. Similarly, let p(τ) = T t=1 p(a t |s t) be the prior distribution over the trajectory, and r(τ) be the return. We seek to maximize the following KL-regularized objective: Since, we can see that this is equivalent to maximizing the following expected value function of the policy π θ at the action level: The two terms introduced in Eq. 6 have clear motivations. The p(a|s) term rewards the model for choosing actions that have high probability under the prior, biasing the model to state-action pairs that are likely to be in the batch. The − log π(a|s) term is analogous to entropy regularization. Maintaining diversity in the action space through entropy regularization is important for generative models like dialog systems, which are known to collapse to an uninteresting, small number of repeated samples (a). Re-stating Eq. 6 as an entropy-regularized Q-function, we obtain: One can derive a soft version of the entropy-regularized Q-function that uses a Boltzmann distribution to estimate future reward . We refer to it as a Ψ-function following previous work , which derived this function as a generalization of the Ψ-learning proposed by . The optimal Ψ-function and policy are: Because it avoids taking a hard max over noisy estimates, Ψ-learning leads to less overestimation of future reward . This improves learning through more stable temporal-difference (TD) updates. Thus, it may be especially useful in the BRL setting for reducing optimism in the face of uncertainty. The Way Off-Policy (WOP) algorithm combines Monte Carlo (MC) target estimation, Ψ-learning, and KL-control from a pre-trained prior. To demonstrate the effectiveness of these techniques, we conduct a series of experiments in traditional RL tasks using the OpenAI gym . Here we show for the CartPole-v0 environment; more are available in the Appendix. We first train an online Qlearning Behavior policy, and store all (s, a, r, s) experience samples into a replay buffer. We use this buffer to train a prior model of p(a|s) using a Variational Auto-encoder (VAE) (details in Appendix). This model is used as a part of both the DBCQ and WOP algorithms. We can use the prior for imitation learning, by sampling actions directly from p(a|s) to obtain Behavioral Cloning (BC). We benchmark all of these techniques against vanilla Q-learning on the batch data (Batch Q). We experiment with four different conditions which vary the quality of the Behavior policy and the replay buffer data: a) Full buffer: all experience samples experienced during online training are used for offline learning; b) Concurrent: the offline learning algorithms see a sliding window of experience samples in the same order that the online learner experienced them; c) Expert demonstrator: the buffer only contains experience generated by a fully trained online learner; and d) Noisy demonstrator: the online learner has a high probability of acting randomly (= 0.3) and is thus a bad model of the optimal policy. Figure 2 shows the . Across conditions, we see that WOP is able to outperform Batch Q, imitation learning (BC), DBCQ, and the original behavior policy. As expected, Imitation learning (BC) underperforms other techniques when the batch contains noisy or inexpert experience samples. However, when the batch contains only expert trajectories, Batch Q fails to learn, because the batch does not cover the full state-action space well, increasing extrapolation error (as illustrated in Figure 1). DBCQ matches or outperforms BC and Batch Q in all scenarios. However, because DBCQ acts by sampling from p(a|s) as learned by the BC model, its performance suffers when the batch data is noisy or imperfect. In contrast, WOP is able to learn to trade-off staying close to the prior and obtaining higher reward, and consistently outperforms all other algorithms in this environment. Here, we tackle the problem of training an open-domain dialog model from human feedback. We consider human interaction to represent the'environment'. The response of a human to the bot's utterance is used to compute a reward signal to train the model. The state is the conversation history, composed of a series of conversation turns or utterances, u 1...t, where each utterance is composed of vocabulary tokens. The model attempts to construct a response utterance u π t+1 = [a 1, a 2, ..., a n] by iteratively choosing an action a i as the next token. Applying RL to dialog generation is challenging due to the large state-action space. The number of tokens in the vocabulary of our pre-trained model is 20,000, making the action space very high-dimensional; this further compounds the problem of extrapolation error. We trained over 40 dialog models with different architectures (e.g. Serban et al. (2017b) ), on different datasets, generating models that varied significantly in terms of the distribution of language they learned. We deployed these models to users via a web server that hosts neural network dialog models on GPU for fast, real-time inference: https://neural.chat. The code for the models and the server is available in open-source at <redacted>. Using the server, we collected a batch of human interaction data containing 14232 pairs of user input and agent response. Because learning language online from humans on the internet can in inappropriate behavior (see), learning offline using BRL is imperative. The batch data was used to train the RL models as described in Section 3. Here, we use a pre-trained language model to estimate p(a|s). We also initialize the weights of the Q-network and target Q-network are from the pre-trained model, to combat extrapolation error. The trained RL models were then re-deployed to the web. We recruited 90 Mechanical Turk workers to provide a total of 718 7-point Likert scale ratings of the bots' quality, fluency, diversity, contingency (relatedness), and empathy, after interacting with each bot for at least 3 turns. Participants also had the option to provide explicit feedback through upvoting or downvoting a particular utterance within the interface. We sum these manual votes to create an overall votes score. We note that using this platform to test our models "in the wild" with humans represents a more meaningful test of generalization than testing an RL model in the same limited (game) environment in which it was trained, since humans are not restricted in the text they can type as input to the model. We seek to improve a dialog model's ability to engage in natural conversation with a human by learning from the signals implicit in the way that the human responds. Rather than having the human manually label good performance -which we show in this work does not scale -the agent should recognize informative cues within the user's responses, like sentiment, and the amount of time they spend chatting. Essentially, we want to create an agent that is intrinsically motivated to produce positive reactions in its human conversation partner. To this end, we reward the model for: 1) eliciting positive sentiment, 2) eliciting longer conversations and more words typed (a sign of engagement), 3) eliciting laughter (in the form of typed 'ha's), 4) high semantic similarity between the human input and bot response, and 5) asking questions, since this is an important active listening skill . The total reward given to the agent is a combination of these, with details (and coefficients) in the Appendix. Note that the first 4 types of rewards depend on eliciting positive responses from a human user; we call these the implicit human reward. The 5th reward is easily exploitable by the agent itself. These rewards were designed and extracted post-hoc from the batch of human data, and thus learning from them is only possible with effective batch RL, since they had no effect on the policies used to generate the batch. Table 2: Purely reward-maximizing methods like Batch Q (left) diverge away from realistic language (saying phrases like "where did you say to me?") in order to trivially exploit the reward function by asking a question every turn, and using the maximum number of tokens in every sentence. In contrast, KL-control methods (right) output plausible language by staying close to the prior, but shift to using polite, cheerful language to maximize implicit human reward. [To compare models, we not only look at human users' ratings and votes, but also consider the automatic signals detectable from the text itself. This implicit human reward metric aggregates the measures listed in items 1-4 in Section 5.1, and measures the ability to elicit positive responses from a human. Table 1 shows the of the human evaluation, comparing WOP to ablations of itself, Batch Q, and DBCQ. MC Target Q estimation leads to modest improvements in votes and human reward, but does not improve ratings. Using Ψ-learning improves all three. However, the most notable difference in performance comes from KL-control. The KL-control models show substantial gains over the baseline models across both ratings and human reward. We perform a oneway analysis of variance (ANOVA) comparing the KL-control models to the Batch Q baselines and DBCQ on the total human rating score, and find that the KL-control models are significantly better, F (x) = 4.781, p <.05. This validates the hypothesis that KL-control with a strong, pre-trained prior can be used to improve batch RL. As shown in Figure 3, without KL-regularization, the baseline RL models diverge quickly and continuously from the prior, losing information about realistic sequences. This helps explain the poor performance of DBCQ in Table 1. The underlying Q-network in DBCQ does not directly integrate the prior. As Q-learning causes the model to diverge from the prior, the Q-estimates of language generated according to the prior become unrealistic, and Eq. 4 selects unrealistic actions. This in highly'diverse' (random) generated utterances. Although DBCQ performed well in simple domains in Section 4, it does not scale effectively to dialog. The pre-trained prior may be especially important in a generative domain like dialog, where the true reward function is unknown, and so purely maximizing a heuristic reward may lead to lower quality conversations. Table 2 shows examples of conversations with a Batch Q and KL-control model. Because the Batch Q model has no incentive to stay close to realistic language, it learns to exploit the reward by asking a question and outputting the maximum number of tokens every utterance. These sentences contain implausible phrases that do not represent realistic language (e.g. "where did you say to me?"). In contrast, the KL-control model uses fluent language, but shifts its distribution towards cheerful and polite speech, presumably because this is what led to positive human responses in the batch data. In fact, we noticed that all models trained with the implicit human rewards described in Section 5.1 learned to use more cheerful and supportive language. Therefore, we create post-hoc metrics to measure this effect (see the Appendix for details). Figure 4 shows how these metrics, as well as the implicit rewards, differ across models. Without KLcontrol, baseline methods like Batch Q exploit simple rewards like asking questions at the expense of realistic language, explaining their poor quality ratings. In contrast, KL-control models learn to rely more on realistic but polite, supportive, and cheerful dialog to elicit higher total human reward. Table 3 presents the of WOP models trained with only a single reward function, ordered from lowest to highest quality. Notably, extracting multiple different reward functions post-hoc from a batch of data and training on these independently is only possible with effective BRL. Investigating which rewards presented are most critical to achieving high-quality conversations with humans, we note that maximizing positive and minimizing negative sentiment in the user turns out to lead to the highest quality bot. This underscores the importance of affective signals as cues for good conversation. Bots trained on the manual upvotes and downvotes provided by users on the utterance level fail to achieve similarly high performance. Even though users were instructed to make use of the vote feature, the task is burdensome, and users did not vote frequently enough to provide a good training signal. This validates the hypothesis that implicit signals of human enjoyment (such as sentiment) are a more scalable way to learn from human preferences. This paper presents the Way Off-Policy (WOP) algorithm, which improves performance when learning off-policy without the possibility to explore -i.e. batch RL (BRL). We are the first to propose using KL-control from a strong prior model pre-trained on data as a way to avoid extrapolation and instability in BRL. Our on traditional RL tasks demonstrate that our WOP algorithm provides performance improvements over state-of-the-art BRL techniques, and the in dialog generation show that KL-control is critical to achieving good performance in this real-world, highdimensional setting. In a generative domain such as dialog, the true reward function is not known, and trivially exploiting the rewards can actually lead to worse performance. Thus, KL-control may be particularly necessary to ensure samples remain realistic and close to the data distribution. We propose several reward functions that could allow an open-domain dialog generation model to learn from rich cues implicit in human interaction, where learning from expressed sentiment was most promising. We find that maximizing implicit rewards leads to better performance than relying on explicit feedback. We hope that the techniques presented here can improve learning with RL from offline data, making it easier to apply RL to safety-critical settings such as human interaction. A APPENDIX The total reward used to train the bots is a combination of the rewards described below, in the following proportions: 0.15682657 * question + 0.13837638 * semantic coherence + 0.15313653 * laughter + 0.14206642 * sentiment transition + 0.14206642 * sentiment + 0.14760148 * words elicited + 0.1199262 * conversation length. To compute sentiment on short texts like conversation utterances, we leverage a state-of-the-art sentiment-detection model, which was trained on a massive amount of Twitter data to predict the emojis in tweets . Transfer learning from this model to other tasks showed that it was able to significantly outperform a series of sentiment, irony, and sarcasm benchmarks. This DeepMoji model outputs a probability distribution over 64 most-frequently used emojis as shown in Figure 5. After observing the performance of the model in detecting users' emotions in the domain of online chat, we define a set of weights over the emojis and calculate the weighted sum over an emotion embedding vector to derive a sentiment reward which is higher for positive sentiment and lower for negative sentiment. These weights are shown in Figure 5 (b). We also compute a sentiment-transition reward using the same score based on whether the peak positive sentiment occurred later in the conversation than the peak negative sentiment, reasoning that sentiment should improve over the course of the conversation. Based on prior work , we use the number of turns in the conversation as an indicator of the quality of the bot's performance. To distribute this reward over every utterance in the conversation, we take the total conversation length N, and compute the discounted reward for utterance n < N as γ N −n N. We also reward each utterance with the number of words in the user's response, which we refer to as the words elicited. Laughter has been shown to be very important to human affiliation and solidarity . Therefore, we detect the number of occurrences of the string'ha' in the user's response, and use this as a reward. Interestingly, we find that bots trained to maximize user laughter learn to be extremely supportive and cheerful compared to other bots (for definitions of supportive and cheerful, see Section ??). Language style matching has been shown to be a strong predictor of relationship initiation and stability . While it would be ideal if our chatbots could intelligently adapt their conversation style to a new user, in reality most baseline dialog models struggle to maintain topic coherence, even over a few utterances (for an analysis of this effect, see ). Therefore we reward semantic similarity between the user's input and the bot's response, to encourage the bot to stay on topic and produce reasonable answers. This score is computing by leveraging a state-of-the-art sentence embedding model , and penalizing distance in embedding space. Asking questions is an important listening skill, and is linked to conversation management, attentiveness, and responsiveness . Therefore, we give the bot a reward of 0.5 if the utterance contains a question word (how, what, where, why, when, who), and an additional 0.5 if it contains a question mark. After training the bots on these rewards, we noticed a shift in the distribution of their language towards more polite, cheerful, and supportive speech. Therefore, we designed post-hoc metrics to measure these qualities, which are based on counting whether a subset of phrases is present in an utterance. Politeness phrases: if I may; may I; please; thanks; no worries; if you don't mind; have a great day; I'm sorry. Supportive phrases: you're right; you are right; you're not alone; you are not alone; congrats; that's a good idea; that is a good idea; you'll be fine; you will be fine; you'll be okay; you will be okay; it will get better; sorry you're going through; sorry you are going through; if it makes you feel better; if it makes you feel any better; keep your head up; keep it up; I'm in a similar situation; I am in a similar situation; you'll get it; you will get it; happy for you; I'm in the same boat; I am in the same boat; if you feel like you need to vent. Cheerful phrases: nice to hear; happy; excited; really nice; glad; the best; great; good time; looking forward; beautiful. All Q-networks shared the same underlying architecture: three fully-connected layers of size, with ReLU activation between. The model of p(a|s) was learned with a Variational Autoencoder which attempted to reconstruct the next state given the current state, p(s |s), using a meansquared error loss. Both the encoder and decoder were made up of two linear layers with 750 neurons each. The latent dimension of the VAE was size 256. The next action, was predicted from the latent embedding z, meaning the model learned three functions: z = f e (s), s = f d (z), and a = f a (z) All models were trained with the Adam optimizer. For each experiment, we ran 50 trials of each model with a different random seed each time. The Behavior policy was trained for a total of 20,000 steps in the environment, so in the Full buffer condition offline agents saw 20,000 experience samples. The Behavior policy typically converged before 10,000 steps, so in the Expert demonstrator condition the offline agents received the last 10,000 experience samples from the trained agent. In the Concurrent condition, offline agents saw a moving window of 1000 samples, since the online learner only used the most recent 1000 samples in the buffer for learning. The learning rate was.001, γ =.99, and decayed linearly from 1.0 to.01 over 2000 steps. The KL-constraint was computed as D KL [q(τ)||p(τ)] = α log p(a|s) − β log π(a|s), where α = 0.5 and β = 0.1. DBCQ sampled n = 2 actions before selecting the best action based on the maximum Q-value; note that in this environment there are only 2 actions. The underlying architecture of the language models employed for this work is a Variational Hierarchical Recurrent Encoder Decoder (VHRED) (b), with additional knowledge distillation to improve the model's ability to track the sentiment and semantics of the conversation, as proposed by. The language models were originally trained on two datasets: movie dialogs and a dataset scraped from reddit.com/r/casual_conversation . The RL models were initialized with the weights of the best model trained on the Reddit dataset. RL models were trained for between 800 and 1000 batches of data, where the batch size was fixed at 32. Early stopping was used to determine the number of training iterations of the best checkpoint. All other hyperparameters were shared between RL models, and were as follows: discount γ = 0.5, weight placed on RL reward vs. KL-divergence term c = 2, number of Monte Carlo samples of the Target Q-network M = 5, target network update rate α =.005, learning rate r =.0001. We used a smooth L1 loss function to approximate the Q-values, and clipped gradients at a value of 1.0. The underlying parameters of the VHRED model were as follows: Context RNN hidden size = 1000, decoder hidden size = 1250, encoder hidden size = 1250, z embedding size = 600, gradient clip = 1.0, dropout d = 0.2. The maximum conversation length was fixed at 5 utterances (context from more than 5 utterances ago was discarded), and the maximum sentence length was 30 tokens. We also added layers to the Context RNN and regularized it to be able to predict the semantic content of the input utterance using a form of knowledge distillation from a state-ofthe-art sentence-embedding model . There were 2 additional feedforward semantic prediction prediction layers of size 128, which used ReLu activation. We ran additional experiments in another standard OpenAI gym environment, Acrobot-v1; Figure 6 shows the . The experimental setup and hyperparameters remained the same as in the first experiment, except that we used a smaller VAE (the encoder and decoder had only one layer of size 256 each, and the latent dimension was 64), the weight on the prior was α = 0.5 and the entropy term was β = 0.1, and we used the Q-learning loss rather than Ψ-learning. Figure 7: Normalized reward scores obtained by models trained with respect to different rewards. We see that the bot trained to ask questions is easily able to exploit this reward, and similarly the bot trained to elicit positive sentiment does so successfully. For the rest of the bots, the relationship is less clear. For example, the bot trained to elicit laughter becomes the most supportive and cheerful, while the bot trained to elicit more words is very polite. Figure 7 shows the normalized reward scores obtained bots trained with respect to different rewards. We see that in testing the bots in the wild, with a new set of users, bots trained to optimize certain types of human response (e.g. words elicited) do not perform best on this task. We hypothesize this is because the relatively small size of batch date we were able to collect (≈ 14, 000 utterances) does not give the bots enough information about how to elicit long responses from users. In addition to the comparisons between RL models, we also benchmarked the RL methods against the MLE prior. While we found that the KL-control methods exceed the prior in obtaining high implicit reward signals from humans (e.g. eliciting positive sentiment, laughter, more words, etc), we found that this did not lead to them being rated significantly higher in quality by human judges. Even though the RL bots successfully learned to conduct conversation to optimize these implicit signals, we believe the fact that this did not lead to higher quality ratings highlights the idea that the reward functions we are optimizing do not fully cover what it means to have a high quality conversation with a human user. We note that these rewards are only a first step, and hope other researchers will be able to use the techniques proposed in this paper to learn from better measures of human enjoyment. To collect data from humans interacting with our bots, we built https://neural.chat, a platform for hosting deep neural network dialog models online on GPU for fast, real-time inference. Figure 8) shows an example of the interface, in which users are able to rate the bots after talking to them for at least three turns. The server was hosted on a Google Cloud Platform virtual instance with 64GB of RAM and a NVIDIA Tesla P100 graphics card. The backend was a Django program being served by NGINX and uWSGI. For simplicity, we opted to have the Django process import the chatbots into the same Python process as Django, rather than have the two connect to each other via other means such as sockets. This configuration decreased development time and increased reliability, but it would need to be revisited if the server needed to scale several orders of magnitude past what was required for this study. The current configuration was still able to support hundreds of simultaneous users and host more than 30 bots concurrently. The chatbots were kept in a separate project from the Django project and maintained separately from the server code. Each chatbot extended an abstract class that defined key methods for the Django program to use, and was registered to a globally accessible dictionary via a decorator. The Django project was provided the path to the Chatbots project in its PYTHONPATH, so it could import the dictionary in which all the chatbot objects had been registered and use that to dynamically determine which chatbots were available and to access them in its views. It is important to note that the chatbots used PyCUDA, and PyCUDA does not work in a multiprocessing environment. Because of this, uWSGI needed to be configured to only have one python process and to disable any attempt at multiprocessing. Furthermore, the chatbots required substantial startup times, so all chatbots are kept in memory at all times in the Django process. In order to keep all the chatbots in memory concurrently, we needed a very high amount of RAM on our server and opted for a 64GB virtual instance, and a GPU with 16GB RAM. This combination of CUDA to run the chatbots on the GPU with a high amount of RAM to keep all bots in memory at the same time ed in incredibly fast server response times, with effectively no increase in response time when using the bots in requests compared to requests that did not. For further information and instructions on server configuration, please read the server documentation available at ¡URL redacted for anonymity¿. We hope that this platform will allow others to host their own bots and evaluate them in an interactive setting.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that KL-control from a pre-trained prior can allow RL models to learn from a static batch of collected data, without the ability to explore online in the environment.

Represent the short summary for finding the corresponding scientific article (long)

The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.