Question: represent the article for finding a single-sentence tldr

Question: This paper introduces R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce a suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration. Reinforcement learning from demonstrations has proven to be an effective strategy for attacking problems that require sample efficiency and involve hard exploration. For example,, and Salimans and Chen (2018b) have shown that RL with demonstrations can address the hard exploration problem in Montezuma's Revenge. Večerík et al., and have demonstrated similar in robotics. Many other works have shown that demonstrations can accelerate learning and address hard-exploration tasks (e.g. see ; ;). In this paper, we attack the problem of learning from demonstrations in hard exploration tasks in partially observable environments with highly variable initial conditions. These three aspects together conspire to make learning challenging: 1. Sparse rewards induce a difficult exploration problem, which is a challenge for many state of the art RL methods. An environment has sparse reward when a non-zero reward is only seen after taking a long sequence of correct actions. Our approach is able to solve tasks where standard methods run for billions of steps without seeing a single non-zero reward. 2. Partial observability forces the use of memory, and also reduces the generality of information provided by a single demonstration, since trajectories cannot be broken into isolated transitions using the Markov property. An environment has partial observability if the agent can only observe a part of the environment at each timestep. 3. Highly variable initial conditions (i.e. changes in the starting configuration of the environment in each episode) are a big challenge for learning from demonstrations, because the demonstrations can not account for all possible configurations. When the initial conditions are fixed it is possible to be extremely efficient through tracking ); however, with a large variety of initial conditions the agent is forced to generalize over environment configurations. Generalizing between different initial conditions is known to be difficult . Our approach to these problems combines demonstrations with off-policy, recurrent Q-learning in a way that allows us to make very efficient use of the available data. In particular, we vastly outperform behavioral cloning using the same set of demonstrations in all of our experiments. Another desirable property of our approach is that our agents are able to learn to outperform the demonstrators, and in some cases even to discover strategies that the demonstrators were not aware of. In one of our tasks the agent is able to discover and exploit a bug in the environment in spite of all the demonstrators completing the task in the intended way. Learning from a small number of demonstrations under highly variable initial conditions is not straight-forward. We identify a key parameter of our algorithm, the demo-ratio, which controls the proportion of expert demonstrations vs agent experience in each training batch. This hyper-parameter has a dramatic effect on the performance of the algorithm. Surprisingly, we find that the optimal demo ratio is very small (but non-zero) across a wide variety of tasks. The mechanism our agents use to efficiently extract information from expert demonstrations is to use them in a way that guides (or biases) the agent's own autonomous exploration of the environment. Although this mechanism is not obvious from the algorithm construction, our behavioral analysis confirms the presence of this guided exploration effect. To demonstrate the effectiveness of our approach we introduce a suite of tasks (which we call the Hard-Eight suite) that exhibit our three targeted properties. The tasks are set in a procedurally-generated 3D world, and require complex behavior (e.g. tool use, long-horizon memory) from the agent to succeed. The tasks are designed to be difficult challenges in our targeted setting, and several state of the art methods (themselves ablations of our approach) fail to solve them. The main contributions of this paper are, firstly we design a new agent that makes efficient use of demonstrations to solve sparse reward tasks in partially observed environments with highly variable initial conditions. Secondly, we provide an analysis of the mechanism our agents use to exploit information from the demonstrations. Lastly, we introduce a suite of eight tasks that support this line of research. Figure 1: The R2D3 distributed system diagram. The learner samples batches that are a mixture of demonstrations and the experiences the agent generates by interacting with the environment over the course of training. The ratio between demos and agent experiences is a key hyper-parameter which must be carefully tuned to achieve good performance. We propose a new agent, which we refer to as Recurrent Replay Distributed DQN from Demonstrations (R2D3). R2D3 is designed to make efficient use of demonstrations to solve sparse reward tasks in partially observed environments with highly variable initial conditions. This section gives an overview of the agent, and detailed pseudocode can be found in Appendix A. The architecture of the R2D3 agent is shown in Figure 1. There are several actor processes, each running independent copies of the behavior against an instance of the environment. Each actor streams its experience to a shared agent replay buffer, where experience from all actors is aggregated and globally prioritized ) using a mixture of max and mean of the TD-errors with priority exponent η = 1.0 as in. The actors periodically request the latest network weights from the learner process in order to update their behavior. In addition to the agent replay, we maintain a second demo replay buffer, which is populated with expert demonstrations of the task to be solved. Expert trajectories are also prioritized using the scheme of. Maintaining separate replay buffers for agent experience and expert demonstrations allows us to prioritize the sampling of agent and expert data separately. The learner process samples batches of data from both the agent and demo replay buffers simultaneously. A hyperparameter ρ, the demo ratio, controls the proportion of data coming from expert demonstrations versus from the agent's own experience. The demo ratio is implemented at a batch level by randomly choosing whether to sample from the expert replay buffer independently for each element with probability ρ. Using a stochastic demo ratio in this way allows us to target demo ratios that are smaller than the batch size, which we found to be very important for good performance. The objective optimized by the learner uses of n-step, double Q-learning (with n = 5) and a dueling architecture (; . In addition to performing network updates, the learner is also responsible for pushing updated priorities back to the replay buffers. In each replay buffer, we store fixed-length (m = 80) sequences of (s,a,r) tuples where adjacent sequences overlap by 40 time-steps. The sequences never cross episode boundaries. Given a single batch of trajectories we unroll both online and target networks on the same sequence of states to generate value estimates with the recurrent state initialized to zero. Proper initialization of the recurrent state would require always replaying episodes from the beginning, which would add significant complexity to our implementation. As an approximation of this we treat the first 40 steps of each sequence as a burn-in phase, and apply the training objective to the final 40 steps only. An Hard-Eight task suite. In each task an agent must interact with objects in its environment in order to gain access to a large apple that provides reward. The 3D environment is also procedurally generated so that every episode the state of the world including object shapes, colors, and positions is different. From the point of view of the agent the environment is partially observed. Because it may take hundreds of low-level actions to collect an apple the reward is sparse which makes exploration difficult. alternative approximation would be to store stale recurrent states in replay, but we did not find this to improve performance over zero initialization with burn-in. Exploration remains one of the most fundamental challenges for reinforcement learning. So-called "hard-exploration" domains are those in which rewards are sparse, and optimal solutions typically have long and sparsely-rewarded trajectories. Hard-exploration domains may also have many distracting dead ends that the agent may not be able to recover from once it gets into a certain state. In recent years, the most notable such domains are Atari environments, including Montezuma's Revenge and Pitfall . These domains are particularly tricky for classical RL algorithms because even finding a single non-zero reward to bootstrap from is incredibly challenging. A common technique used to address the difficulty of exploration is to encourage the agent to visit under-explored areas of the state-space . Such techniques are commonly known as intrinsic motivation or count-based exploration . However, these approaches do not scale well as the state space grows, as they still require exhaustive search in sparse reward environments. Additionally, recent empirical suggest that these methods do not consistently outperform -greedy exploration (Taïga et al., 2019). The difficulty of exploration is also a consequence of the current inability of our agents to abstract the world and learn scalable, causal models with explanatory power. Instead they often use low-level features or handcrafted heuristics and lack the generalization power necessary to work in a more abstract space. Hints can be provided to the agent which bias it towards promising regions of the state space either via reward-shaping or by introducing a sequence of curriculum tasks . However, these approaches can be difficult to specify and, in the case of reward shaping, often lead to unexpected behavior where the agent learns to exploit the modified rewards. Another hallmark of hard-exploration benchmarks is that they tend to be fully-observable and exhibit little variation between episodes. Nevertheless, techniques like random no-ops and "sticky actions" have been proposed to artificially increase episode variance in Atari , an alternative is to instead consider domains with inherent variability. Other recent work on the Obstacle Tower challenge domain is similar to our task suite in this regard. Reliance on determinism of the environment is one of the chief criticisms of imitation leveled by , who offers a valuable critique on, and Salimans and Chen (2018a). In contrast, our approach is able to solve tasks with substantial per-episode variability. GAIL is another imitation learning method, however GAIL has never been successfully applied to complex partially observable environments that require memory. Even the maze task inŻołna et al. has distinguishable rooms, uses a single layout across all episodes, and as a does not require a recurrent policy or discriminator. High-level steps necessary to solve the Baseball task. Each step in this sequence must be completed in order, and must be implemented by the agent as a sequence of low level actions (no option structure is available to the agent). The necessity of completing such a long sequence of high level steps makes it unlikely that the task will ever be solved by random exploration. Note each step involves interaction with physical objects, shown in bold. To address the difficulty of hard exploration in partially observable problems with highly variable initital conditions we introduce a collection of eight tasks, which exhibit these properties. Due to the generated nature of these tasks and the rich form of interaction between the agent and environment, we see greatly increased levels of variability between episodes. From the perspective of the learning process, these tasks are particularly interesting because just memorizing an open loop sequence of actions is unlikely to achieve even partial success on a new episode. The nature of interaction with the environment combined with a limited field of view also necessitates the use of memory in the agent. All of the tasks in the Hard-Eight task suite share important common properties that make them hard exploration problems. First, each task emits sparse rewards-in all but one task the only positive instantaneous reward obtained also ends the episode. The visual observations in each task are also first-person and thus the state of the world is only ever partially observed. Several of the tasks are constructed to ensure that that it is not possible to observe all task relevant information simultaneously. Finally, each task is subject to a highly variable initial conditions. This is accomplished by including several procedural elements, including colors, shapes and configurations of task relevant objects. The procedural generation ensures that simply copying the actions from a demonstration is not sufficient for successful execution, which is a sharp contrast to the the case of Atari . A more detailed discussion of these aspects can be found in Appendix B and videos of agents and humans performing these tasks can be found at https://bit.ly/2mAAUgg. Each task makes use of a standardized avatar with a first-person view of the environment, controlled by the same discretized action space consisting of 46 discrete actions. In all tasks the agent is rewarded for collecting apples and often this is the only reward obtained before the episode ends. A depiction of each task is shown in Figure 2. A description of the procedural elements and filmstrip of a successful episode for each task is provided in Appendix B. Each of these tasks requires the agent to complete a sequence of high-level steps to complete the task. An example from the task suite is shown in Figure 3. The agent must: find the bat, pick up the bat, knock the ball off the plinth, pick up the ball, activate the sensor with the ball (opening the door), walk through the door, and collect the large apple. In this section we discuss the baselines and ablations we use to compare against our R2D3 agent in the experiments. We compare to Behavior Cloning (a common baseline for learning from demonstrations) as well as two ablations of our method which individually remove either recurrence or demonstrations from R2D3. The two ablations correspond to two different state of the art methods from the literature. Behavior Cloning BC is a simple and common baseline method for learning policies from demonstrations . This algorithm corresponds to a supervised Frames of size 96x72 are fed into a ResNet, the output is then augmented by concatenating the previous action at−1, previous reward rt−1, and other proprioceptive features ft, such as accelerations, whether the avatar hand is holding an object, and the hand's relative distance to the avatar. learning approach to imitation learning which uses only expert trajectories as its training dataset to fit a parameterized policy mapping states to actions. For discrete actions this corresponds to a classification task, which we fit using the cross-entropy loss. If the rewards of trajectories in the training dataset are consistently high, BC is known to outperform recent batch-RL methods . To enable fair comparison we trained our BC agent using the same recurrent neural network architecture that we used for our R2D3 algorithm (see Figure 4). The first ablation we consider is to remove demonstrations from R2D3. This corresponds to setting the demo ratio (see Figure 1) to ρ = 0. This special case of R2D3 corresponds exactly to the R2D2 agent of , which itself extends DQN to partially observed environments by combining it with recurrence and the distributed training architecture of Ape-X DQN. This ablation is itself state of the art on Atari-57 and DMLab-30, making it an extremely strong baseline. No Recurrence The second ablation we consider is to replace the recurrent value function of R2D3 with a feed-forward reactive network. We do this separately from the no demonstrations ablation, leaving the full system in Figure 1 in tact, with only the structure of the network changed. If we further fix the demo ratio to ρ = 0.25 then this ablation corresponds to the DQfD agent of , which is competitive on hard-exploration Atari environments such as Montezuma's Revenge. However, we do not restrict ourselves to ρ = 0.25, and instead optimize over the demo ratio for the ablation as well as for our main agent. We evaluate the performance of our R2D3 agent alongside state-of-the-art deep RL baselines. As discussed in Section 5, we compare our R2D3 agent to BC (standard LfD baseline) R2D2 (off-policy SOTA), DQfD (LfD SOTA). We use our own implementations for all agents, and we plan to release code for all agents including R2D3. For each task in the Hard-Eight suite, we trained R2D3, R2D2, and DQfD using 256 -greedy CPUbased actors and a single GPU-based learner process. Following, the i-th actor was assigned a distinct noise parameter i ∈ [0.4 8,0.4] where each i is regularly spaced in log 0.4 space. For each of the algorithms their common hyperparameters were held fixed. Additionally, for R2D3 and DQfD the demo ratio was varied to study its effect. For BC we also varied the learning rate independently in a vain attempt to find a successful agent. All agents act in the environment with an action-repeat factor of 2, i.e. the actions received by the environment are repeated twice before passing the observation to the agent. Using an action repeat of 4 is common in other domains like Atari ; however, we found that using an action repeat of 4 made the Hard-Eight tasks too difficult for our demonstrators. Using Figure 5: Reward vs actor steps curves for R2D3 and baselines on the Hard-Eight task suite. The curves are computed as the mean performance for the same agent across 5 different seeds per task. Error regions show the 95% confidence interval for the mean reward across seeds. Several curves overlap exactly at zero reward for the full range of the plots. R2D3 can perform human-level or better on Baseball, Drawbridge, Navigate Cubes and Wall Sensor. R2D2 could not get any positive rewards on any of the tasks. DQfD and BC agents occasionally see rewards on Drawbridge and Navigate Cubes tasks, but this happens rarely enough that the effect is not visible in the plots. Indicators mark analysis points in Section 6.3. an action repeat of 2 allowed us to strike a compromise between ease of demonstration (high action repeats prohibiting smooth and intuitive motion) and ease of learning for the agents (low action repeats increase the number of steps required to complete the task). Figure 4 illustrates the neural network architecture of the different agents. As much as possible we use the same network architecture across all agents, deviating only for DQfD, where the recurrent head is replaced with an equally sized feed-forward layer. We briefly outline the training setup below, and give an explicit enumeration of the hyperparameters in Appendix C. For R2D3, R2D2 and DQfD we use the Adam optimizer with a fixed learning rate of 2×10 −4. We use hyperparameters that are shown to work well for similar environments. We use distributed training with 256 parallel actors, trained for at least 10 billion actor steps for all tasks. For the BC agent the training regime is slightly different, since this agent does not interact with the environment during training. For BC we also use the Adam optimizer but we additionally perform a hyperparameter sweep over learning rates {10 −5,10 −4,10 −3}. Since there is no notion of actor steps in BC we trained for 500k learner steps instead. During the course of training, an evaluator process periodically queries the learner process for the latest network weights and runs the ing policy on an episode, logging both the final return and the total number of steps (actor or learner steps, as appropriate) performed at the time the of evaluation. We collected a total of 100 demonstrations for each task spread across three different experts (each expert contributed roughly one third of the demonstrations for each task). Demonstrations for the tasks were collected using keyboard and mouse controls mapped to the agent's exact action space, which was necessary to enable both behaviour cloning and learning from demonstrations. We show statistics related to the human demonstration data which we collected from three experts in Table 1. In Figure 5, we report the return against the number of actor steps, averaged over five random initializations. We find that none of the baselines succeed in any of the eight environments. Meanwhile, R2D3 learns six out of the eight tasks, and reaches or exceeds human performance in four of them. The fact that R2D3 learns at all in this setting with only 100 demonstrations per task demonstrates the ability of the agent to make very efficient use of the demonstrations. This is in contrast to BC and DQfD which use the same demonstrations, and both fail to learn a single task from the suite. All methods, including R2D3, fail to solve two of the tasks: Remember Sensor and Throw Across. These are the two tasks in the suite that are most demanding in terms of memory requirements for the agent, and it is possible that our zero-initialization with burn-in strategy for handling LSTM states in replay does not give R2D3 sufficient context to complete these tasks successfully. Future work should explore the better handling of recurrent states as a possible avenue towards success on these tasks. R2D3, BC, and DQfD receive some negative returns on Remember Sensor, which indicates that the agents navigate down the hallway and walks over penalty sensors. R2D3 performed better than our average human demonstrator on Baseball, Drawbridge, Navigate Cubes and the Wall Sensor tasks. The behavior on Wall Sensor Stack in particular is quite interesting. On this task R2D3 found a completely different strategy than the human demonstrators by exploiting a bug in the implementation of the environment. The intended strategy for this task is to stack two blocks on top of each other so that one of them can remain in contact with a wall mounted sensor, and this is the strategy employed by the demonstrators. However, due to a bug in the environment the strategy learned by R2D3 was to trick the sensor into remaining active even when it is not in contact with the key by pressing the key against it in a precise way. In light of the uniform failure of our baselines to learn on the Hard-Eight suite we made several attempts at training other models on the task suite; however, these attempts were all unsuccessful. For example, we tried adding randomized prior functions to R2D2, but this approach was still unable to obtain reward on any of the Hard-Eight tasks. We also trained an IMPALA agent with pixel control as auxiliary reward to help with exploration, but this approach also failed to learn on any of the tasks we attempted. We omit these from Figure 5, only keeping the most relevant baselines. In our experiments on Hard-Eight tasks (see Figure 5), we did a hyperparameter search and chose the best hyperparameters for each method independently. In this section, we look more closely at how the demo ratio (ρ) affects learning in R2D3. To do this we look at how the success rate of R2D3 across the entire Hard-Eight task suite varies as a function of the demo ratio. The goal of each task in the Hard-Eight suite is to collect a large apple, which ends the episode and gives a large reward. We consider an episode successful if the large apple is collected. An agent that executes many episodes in the environment will either succeed or fail at each one. We consider an agent successful if, after training, at least 75% of its final 25 episodes are successful. Finally, an individual agent with a fixed set of hyperparameters may still succeed or fail depending on the randomness in the environment and the initialization of the agent. We train several R2D3 agents on each tractable task 1 in the Hard-Eight suite, varying only the demo ratio while keeping other hyperparameters fixed at the values used for the learning experiment. We consider four different demo ratios across six tasks, with five seeds for each task (120 trained agents). Figure 6 shows estimates of the success rate for the R2D3 algorithm for each different demo ratio, aggregated across all tasks. We observe that tuning the demo ratio has a strong effect on the success rate across the task suite, and that the best demo ratio is quite small. See Appendix D.3 for further . The typical strategy for exploration in RL is to either use a stochastic policy and sample actions, or to use a deterministic policy and take random actions some small fraction of the time. Given sufficient time both of these approaches will in theory cover the space of possible behaviors, but in practice the amount of time required to achieve this coverage can be prohibitively long. In this experiment, we compare the behavior of R2D3 to the behavior of R2D2 (which is equivalent to R2D3 without demonstrations) on two of the tasks from the Hard-Eight suite. Even very early in training (well before R2D3 is able to reliably complete the tasks) we see many more task-relevant actions from R2D3 than from R2D2, suggesting that the effect of demonstrations is to bias R2D3 towards exploring relevant parts of the environment. In Figure 7 we begin by examining the Push Blocks tasks. The task here is to push a particular block onto a sensor to give access to a large apple, and we examine the behavior of both R2D3 and R2D2 after 5B steps, which is long before R2D3 begins to solve the task with any regularity (see Figure 5). Looking at the distribution of spatial locations for the agents it is clear that R2D2 essentially diffuses randomly around the room, while R2D3 spends much more time in task-relevant parts of the environment (e.g. away from the walls). We also record the total distance traveled by the moveable blocks in the room, and find that R2D3 tends to move the blocks significantly more often than R2D2, even before it has learned to solve the task. In this paper, we introduced the R2D3 agent, which is designed to make efficient use of demonstrations to learn in partially observable environments with sparse rewards and highly variable initial conditions. We showed through several experiments on eight very difficult tasks that our approach is able to outperform multiple state of the art baselines, two of which are themselves ablations of R2D3. We also identified a key parameter of our algorithm, the demo ratio, and showed that careful tuning of this parameter is critical to good performance. Interestingly we found that the optimal demo ratio is surprisingly small but non-zero, which suggests that there may be a risk of overfitting to the demonstrations at the cost of generalization. For future work, we could investigate how this optimal demo ratio changes with the total number of demonstrations and, more generally, the distribution of expert trajectories relative to the task variability. We introduced the Hard-Eight suite of tasks and used them in all of our experiments. These tasks are specifically designed to be partially observable tasks with sparse rewards and highly variable initial conditions, making them an ideal testbed for showcasing the strengths of R2D3 in contrast to existing methods in the literature. Our behavioral analysis showed that the mechanism R2D3 uses to efficiently extract information from expert demonstrations is to use them in a way that guides (or biases) the agent's own autonomous exploration of the environment. An in-depth analysis of agent behavior on the Hard-Eight task suite is a promising direction for understanding how different RL algorithms make selective use of information. A R2D3 Below we include pseudocode for the full R2D3 agent. The agent consists first of a single learner process which samples from both demonstration and agent buffers in order to update its policy parameters. Inputs: replay of expert demonstrations D, replay of agent experiences R, batch size B, sequence length m, and number of actors A. Initialize policy weights θ. Initialize target policy weights θ ← θ. Launch A actors and replicate policy weights θ to each actor. for nsteps do Sample transition sequences (st:t+m,at:t+m,rt:t+m) from replay D with probability ρ or from replay R with probability (1−ρ), to construct a mini-batch of size B. Calculate loss using target network. Perform a gradient descent step to update θ. If t mod ttarget = 0, update the target policy weights θ ← θ. If t mod tactor = 0, replicate policy weights to the actors. end for The agent also consists of A parallel actor processes which interact with a copy of the environment in order to obtain data which is then inserted into the agent buffer. The agents periodically update their parameters to match those being updated on the learner. Sample action from behavior policy a ← π(s) Execute a and observe s and r Store (s,a,s,r) in R until learner finishes. Sparse rewards All of the tasks emit sparse rewards, indeed in all but one task the only positive instantaneous reward obtained also ends the episode successfully. In other words, for standard RL algorithms to learn by bootstrapping, the actors must first solve the task inadvertently, and must do so with no intermediate signal to guide them. Partial observability Visual observations are all first-person, which means that some relevant features of the state of the world may be invisible to the agent simply because they are behind it or around a corner. Some tasks (e.g. Remember Sensor, are explicitly designed so that this is the case). Highly Variable Initial Conditions Many of the elements of the tasks are procedurally generated, which leads to significant variability between episodes of the same task. In particular, the starting position and orientation of the agent are randomized and similarly, where they are present, the shapes, colors, and textures of various objects are randomly sampled from a set of available such features. Therefore a single (or small number of) demonstration(s) is not sufficient to guide an agent to solve the task as it is in the case of DQfD on Atari . Observation specification All of the tasks provide the same observation space. In particular, a visual channel consisting of 96 by 72 RGB pixels, as well as accelerations of the avatar, force applied by the avatar hand on the object, whether if the avatar is holding anything or not, and the distance of a held object from the face of the avatar (zero when there is no held object). Action specification The action space consists of four displacement and four rotation actions, duplicated for coarse and fine-grained movement as well as for movement with and without grasping. The avatar also has an invisible "hand" which can be used to manipulate objects in the environment. The location of the hand is controlled by the avatar gaze direction, plus an additional two actions that control the distance of the hand from the body. A grasped object can be manipulated by six rotation actions (two for each rotational degree of freedom; 40) as well as four additional actions controlling the distance of the hand from the body at coarse and fine speed. Finally there is an independent grasp action (to hold an object without moving), and a no-op action (total 46). Compared to course actions, fine-grained actions in slower movements, allowing the agent to perform careful manipulations. This section gives addition details on each task in our suite including a sequence frames from a successful task execution (performed by a human) and a list of the procedural elements randomized per episode. Videos of agents and humans performing these tasks can be found at https://bit.ly/2mAAUgg. The agent spawns in a small room with a sensor and a key object resting high atop a plinth. The agent must find a stick and use it to knock the key object of the plinth in order to activate the sensor. Activating the sensor opens a door to an adjoining room with a large apple which ends the episode. • Initial position and orientation of the agent • Wall, floor and object materials and colors • Initial position of the stick • Position of plinth The agent spawns at one end of a network of branching platforms separated by drawbridges, which can be activated by touching a key object to a sensor. Activating a drawbridge with a key object destroys the key. Each platform is connected to several drawbridges, but has only one key object available. Some paths through the level have small apples which give reward. The agent must choose the most rewarding path through the level to obtain a large apple at the end which ends the episode. • Initial position and orientation of the agent • Wall, floor, ceiling and object materials and colors • Positions of the small apples throughout the network of ledges The agent spawns on one side of a large room. On the other side of the room on a raised platform there is a large apple which ends the episode. Across the center of the room there is a wall of movable blocks. The agent must dig through the wall of blocks and find a ramp onto the goal platform in order to collect the large apple. • Initial position and orientation of the agent • Wall, floor and object materials and colors The agent spawns in a medium sized room with a recessed sensor in the floor. There are several objects in the room that can be pushed but not lifted. The agent must push a block whose color matches the sensor into the recess in order to open a door to an adjoining room which contains a large apple which ends the episode. Pushing a wrong object into the recess makes the level impossible to complete. • Initial position and orientation of the agent • Wall, floor, object materials and colors • Positions of the objects • Sensor required color The agent spawns near a sensor of a random color. The agent must travel down a long hallway to a room full of blocks and select one that matches the color of the sensor. Bringing the correct block back to the sensor allows access to a large apple which ends the episode. In addition to being far away, traveling between the hallway and the block room requires the agent to cross penalty sensors which incurs a small negative reward. • Initial position and orientation of the agent • Sensor required color • Number of objects in the block room • Position of objects in the block room • Shape and material of the objects in the block room The agent spawns in a U shaped room with empty space between the legs of the U. There are two key objects near the agent spawn point. The agent must throw one of the key objects across the void, and carry the other around the bottom of the U. Both key objects are needed to open two locked doors which then give access to a large apple which ends the episode. • Initial position and orientation of the agent • Wall, floor and object materials and colors • Color and material of the sensors • Initial positions of the two key objects The agent spawns in a small room with a wall mounted sensor and a key object. The agent must pick up the key and touch it to the sensor which opens a door. In the adjoining room there is a large apple which ends the episode. • Initial position and orientation of the agent • Position of the sensor • Position of the key object The agent spawns in a small room with a wall mounted sensor and two key objects. This time one of key objects must be in constant contact with the sensor in in order for the door to remain open. The agent must stack the two objects so one can rest against the sensor, allowing the agent to pass through to an adjoining room with a large apple which ends the episode. An important property of R2D3 is that although the agents are trained from demonstrations, the behaviors they achieve are able to surpass the skill of the demonstrations they were trained from. This can be seen quantitatively from reward curves in Figure 5, where the R2D3 agent surpasses the human baseline performance on four of the eight tasks (e.g. Baseball, Navigate Cubes, Wall Sensor and Wall Sensor Stack). In some of these cases the improved score is simply a matter of executing the optimal strategy more fluently than the demonstrators. For example, this is the case in the Baseball task, where the human demonstrators are handicapped by the fact that the human interface to the agent action space makes it awkward to rotate a held object. This makes picking up the stick and orienting it properly to knock the ball off the plinth into a tricky task for humans, but the agents are able to refine their behavior to be much more efficient (see Figure 8c). The behavior on Wall Sensor is especially interesting, however in this case the agents find a completely different strategy than the human demonstrators by exploiting a bug in the implementation of the environment. The intended strategy for this task is to stack two blocks on top of each other so that one of them can remain in contact with a wall mounted sensor, and this is the strategy employed by the demonstrators. However, due to a bug in the environment it is also possible to trick the sensor into remaining active even when it is not in contact with the key by pressing the key against it in a precise way. The R2D3 agents are able to discover this bug and exploit it, ing in superhuman scores on this task even though this strategy is not present in the demonstrations. We also ran a few additional experiments to get more information about the tasks we did not solve, or solved incorrectly. Videos for these experiments are available at https://bit.ly/2mAAUgg. Remember Sensor This task requires a long memory, and also has the longest episodes length of any task in the Hard Eight suite. In an attempt to mitigate these issues, we trained the agent using a higher action repeat which reduces the episode length, and used stale lstm states instead of zero lstm states which provides information from earlier in the episode. This allows R2D3 to learn policies that display reasonable behavior, retrieving a random block and bringing it back to the hallway. Using this method it can occasionally solve the task. Throw Across The demonstrations collected for this task had a very low success rate of 54%. We attempted to compensate for this by collecting an additional 30 demos. When we trained R2D3 with all 130 demos all seeds solved the task. Wall Sensor Stack The original Wall Sensor Stack environment had a bug that the R2D3 agent was able to exploit. We fixed the bug and verified the agent can learn the proper stacking behavior. In Figure 9, we show the performance of the R2D3 agents for each seed separately. On task such as Drawbridge, Navigate Cubes and Wall Sensor, all seeds take off quite rapidly and they have very low variance for the rewards between different seeds. However, on Wall Sensor Stack task while one seed takes off quite rapidly, and the rest of them are just flat. In Figure 10, we elaborate on Figure 6. For Baseball, Navigate Cubes, Push Blocks, and Wall Sensor Stack, a demo ratio of 1/256 works best. On Drawbridge and Wall Sensor all demo ratios are similarly effective. Figure 7 ). (a) Spatial pattern of exploration behavior for the R2D2 agent over the course of ∼12B steps of training. Each row shows a different random seed; the number of training steps increases from the leftmost column to the rightmost column. There is little variation in how the policy manifests as explorative behavior across seeds and training time. (b) As in (a), for R2D3. Given demonstrations, the policies now show substantial variation across seeds and training time.

We introduce R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions.

We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.


Question: Represent the article for finding a single-sentence TLDR

Question: We investigate the learned dynamical landscape of a recurrent neural network solving a simple task requiring the interaction of two memory mechanisms: long- and short-term. Our show that while long-term memory is implemented by asymptotic attractors, sequential recall is now additionally implemented by oscillatory dynamics in a transverse subspace to the basins of attraction of these stable steady states. Based on our observations, we propose how different types of memory mechanisms can coexist and work together in a single neural network, and discuss possible applications to the fields of artificial intelligence and neuroscience. Recurrent neural networks (RNN) are widely used to carry out tasks that require learning temporal dependencies across several scales. Training RNN's to perform such tasks offers its share of challenges, from well-known exploding and vanishing gradients, to the difficulties of storing, accessing, and forgetting memories BID10 BID1. Viewed as dynamical system, the activity structure of recurrent network state spaces can reveal how networks learn tasks, and can help guide training and architecture design. In this study, we perform a dynamical system analysis of a trained RNN on a simple tasks that requires two types of memory paradigms interact: short-term memory of past inputs and a delayed output during classification. While gating units found in LSTM BID11 and in a variety of other architectures (e.g., BID2 van der) directly aim at addressing these long-scale temporal learning issues, they are always used in conjunction with so-called "vanilla" recurrent units that shoulder the majority of computation. It is not yet well understood how internal network dynamics supported by such circuits combine information from external inputs to solve complex tasks that require remembering information from the past and delaying output changes. On one hand, attractor networks are a known solution to keep finite memories indefinitely BID5. On the other, orthogonal transformations (e.g., identity and rotations) are used to build explicit RNN solutions to recall tasks BID6 BID13 BID0. Indeed, for the well-studied copy task, where a sequence of symbols needs to be outputted after a long delay, it is known that the best solution is to use rotations to store the sequences, much like clocks that align at the time of recall BID3. However, it is unclear how attractor dynamics and orthogonal (rotational) transformations interact when a task requires both long term memory and sequential recall. We explore this situation here. Leveraging slow-point analysis techniques BID12, we uncover how short-term memory tasks with delayed outputs give rise to attractor dynamics with oscillatory transients in low-dimensional activity subspaces. Our uncovers how the boundaries of basins of attractions that are linked to memory attractors interact with transverse oscillatory dynamics to support timed, sequential computations of integrated inputs. This provides novel insights into dynamical strategies to solve complex temporal tasks with randomly connected recurrent units. Moreover, such transient oscillatory dynamics are consistent with periodic activity found throughout the brain BID9, and we discuss the impact of our findings on computations in biological circuits. In this paper we replicate and advance further the study of BID12 on the implementation of memory tasks in artificial RNNs. This simple analysis allows for careful examination of the inner workings of RNNs, which we leverage here to study the interaction of two distinct memory mechanisms in it with the introduction of delayed recall in a memory task. While the original study (i.e., BID12 used continuous time, we used a discrete-time network and trained it with standard optimization algorithms. This is in order to verify if the original findings translate well between the different types of network, and to explore network architectures that are more widely used in the machine learning community. Detailed description of our setup is provided in Section 3.We base our analysis on a densely connected single-layer RNN with a hyperbolic tangent activation function. Specifically, let X be the input signal, S the hidden state of the network and Y the output of the network. We use N = 100 neurons with input matrix W in, output matrix W out, and recurrent connection (i.e., used to update the hidden layer over time) matrix W. Then, the update rule for the hidden states and outputs of the network are defined as: DISPLAYFORM0 For training, we unfold the network in time, up to ten timesteps, and use the standard Adam BID7 optimization algorithm. Only the W and W out matrices were trained. See Appendix A for more details. We characterized the behavior of the neural circuit using tools from dynamical system theory, which enable us to identify internal states and transitions over time in the neural circuit. In particular, we use slow point analysis, developed in BID12, to find approximate fixed points. Formally, let S(t) be the state of the network at time t. We define a Lyapunov function q inspired by the physical formula for potential energy. We adapt this original approach to a discrete-time system as DISPLAYFORM1 The minima of this function indicate the fixed points of the system, and their types (e.g., sink or saddle point) can be determined by the number of positive eigenvalues of the Jacobian matrix at that point. We use a delayed 3-bit flip-flop task to illustrate our findings, as it requires the interaction of two types of memory mechanisms and lends itself to easily interpretable network dynamics. The network has three independent channels with an associated input (W in) and output (W out) neuron each. Each input neuron sends short spikes of magnitude 1 or -1 at random time intervals (Poisson distribution with homogeneous rate λ = 50) and is set at zero otherwise. The output channel must maintain the value of the last input spike sent by its associated input neuron. To study the interaction between long term memory and sequential recall, we add a time delay to this task. This Figure 1. Task definition. The neural network is expected to match and maintain the last input for each channel while preventing crosstalk. For the delayed task definition, the neural network must, in addition, delay its response (and therefore remember the input) during a certain number of time steps by ∆t, illustrated in grey.in turn requires the network to delay its change of internal state upon the reception of a novel input by a period ∆t. Representative examples of both tasks are demonstrated in Figure 1. We begin by replicating the observations from BID12 in our discrete-time setting. We tested the trained network during roughly a hundred thousand steps, during which the network receives approximately 2000 inputs, saving the neurons states and the network output at every step. We then analyzed the ing dynamics, identifying slow and fixed points with slow-point analysis, and projecting dynamics in the first three principal components (PC), i.e., using PCA.As we can see in figure 2a, we do observe the characteristic "cube" discovered in the original research. Here, each vertex represents a certain output state (since there are three binary channels, this means there are eight possible states), with adjacent vertices varying by a single output. The slow point analysis also revealed attractors at the corners of the cube described by the path of the states, saddle points with one unstable dimension on the edges, saddle points with two unstable dimensions on the faces and a saddle point with three unstable dimensions at the center. The attractors implement long-term memory by "trapping" trajectories in the absence of inputs. Each of the eight attractors encodes a specific output state. Saddle-points channel the dynamics during a switch of output state. Their associated stable subspaces form the separatrices between the basins of attraction of the stable fixed points on each side. The configuration of fixed points observed here is also consistent with the findings of BID12.We next turn to the trajectories produced by the network trained to perform the delayed task (see Figure 2b). The position of the "corner" attractors is fairly similar to that of the zero-delay case. However, the steady-state dynamics (a) (b) Figure 2. (a) Dynamics during 3000 time-steps without delay, projected in the first three principal components. Each point represents the network's state at a different time-step and is color-coded according to the effective output value at that time. In the absence of input, the network will maintain its position in phase-space at one of 8 stable fixed points, encoding the current binary combination of the 3 input channels. Injecting an input in the network induces a switch of the network towards different attracting states, and thus, produces distinct outputs. (b) Same as in (a) for dynamics with delayed output. The network must generate the same output as before but delay its output change for ∆t = 8 time-steps.surrounding these corners have changed considerably. One hint about the hidden dynamics implementing sequential recall is the "loops" protruding from the corners of the initial cube and disappearing with the addition of delay. Since we use a tanh activation function that limits outputs between 1 and -1, overshooting the attractor assures that the correct output is expressed immediately after receiving the input, while undershooting it would give a visible transient period. However, in the delayed task, inputs have to be treated in a subspace orthogonal to the output space since no output change must be expressed before the delay has elapsed. This means that the input pulse is now invisible in the output space. With these observations in mind, we would like to further investigate the different hidden dynamics brought by the added delay. Since standard PCA and slow-point analysis failed to sufficiently inform us about this aspect, we now turn to triggered activation averages described in Section 5. For simplicity, we focus at this point on characterizing a single type of transition in the network. Without loss of generality, we chose the transition from state (−1, −1, −1) to state (−1, −1, 1), and searched every instance during testing when the network had to perform this switch. We note that this choice of input switch is arbitrary but fixed, although other switches can also be analyzed in a similar manner. For each instance, we kept a trajectory of RNN states (i.e., hidden-layer activations) starting a few steps before the switch and ending a few steps after it. We then averaged these trajectories over instances, aligned at the switch time, in order to obtain a single short average trajectory (in the hidden-layer activation space) representing the mean activity surrounding the examined switch. PCA projection of this new signal is shown in FIG1.In FIG1, we see a decomposition akin to Fourier modes of a step function, with increasingly rapid oscillations occurring in one forth of a period phase shifted pairs. Here, individual neurons appear to lock into dynamical regimes associated to precise frequencies. From the way the cube is flattened in FIG1, we also conclude that the subspace where the rotational transformations are implemented is mostly orthogonal to the separatrices of the attractors corresponding to the different memorized outputs. A schematic of the transition is presented at figure 3a. We note that such rotating dynamics have indeed been observed previously when implementing sequential recall in artificial neural networks, and it is known that initializing RNNs with orthogonal rotation matrices helps solve sequential recall tasks BID3.The periodic nature of the observed signals motivated us to explore the eigenspectrum of the matrix W (see Figure 4). Indeed, when considering the complex eigenvalues of this matrix, their complex parts correspond to rotating speeds in different subspaces. For comparison, the network without delay has only three eigenvalues that distinguish themselves from the random cloud around zero, which is expected from random Gaussian matrices. These are on or close to the real axis, and slightly higher than one. Considering the hyperbolic tangent activation function applied at every time-step, this indicates three dimensions in which the state compo- The switch from output (-1,-1,-1) to output (-1,-1,1) for the delayed network was studied using a triggered average. The ing signal was then plotted according to the first principal components. For clarity, a cube is used to mark the approximate trajectory of neurons states during the test. Trajectory in the space spanned by the first three principal components. (c) Same trajectory but with the first 9 principal components represented.nents are maintained at about 1. Figure 4. Analysis of the eigenvalues of the recurrent connection matrix W for the studied RNNs. The eigenvalues of the network without delay is displayed to the left and the ones for the 8 timesteps delayed network to the right. In the delayed network, the rotating dynamics are clearly indicated by eigenvalues with large imaginary parts, in addition to the three real eigenvalues corresponding to the long term memory subspace. This additional information is consistent with our interpretation that oscillatory dynamics implement sequential recall, much like orthogonal RNNs do during simple recall tasks BID6 BID13 BID0 BID3. However, in this case they do so in a localized fashion to facilitate interaction with other computation mechanisms implemented by the network. We have seen in this study that long-term memory and sequential recall can be implemented by a simple RNN fairly easily, and in parallel, by acting on different subspaces of the RNN phase space. Specifically, sequential recall is achieved by rotational dynamics localized around the origin, which occur in a subspace orthogonal to the separatrices of the basins of attraction that solve the classification task. Our findings suggest that this population-level periodic activity may serve as a general "precision timing" mechanism that can be combined with distinct, learned computations. Indeed, oscillations enable the introduction of small delays, transverse to low dimensional activity of recurrent neural circuits. An interesting line of future work would be to investigate more thoroughly this mechanism in the presence of distinct computational tasks, such as character-level prediction, or arithmetic operations. We believe that learning a delayed recall in conjunction with any task will lead to generic, emergent oscillations that enable transient dynamics transverse to the subspaces used to perform other computations. It may be possible to leverage this geometric understanding for faster training by initializing networks in a way that promotes transverse rotations. Furthermore, this oscillatory mechanism is consistent with observations of oscillatory dynamics in the brain BID9. Together with the known phenomena whereby neurons in the brain perform tasks with low-dimensional activity patterns, and that the same neurons engage in oscillatory activity when viewed at the population-level, our findings are consistent with a general principle of delayed recall in neural networks, either biological or artificial. The network was a one-layered discrete and densely connected recurrent neural network with a N = 100 neurons. The network is composed of three real-valued matrix, the square matrix W representing the recurrent connection between the neurons, the 3 × 100 W in matrix representing the strength of connections between the input channels and the neurons of the network and the 100 × 3 W out matrix representing the strength of the connection between the neurons of the network and the output neurons. At each neuron, the signal goes through an "activation function", in this case the hyperbolic tangent. A schematic of the neural network is shown on figure 5a. The hyperbolic tangent is often used as the activation function in studies about neural networks as it is a good representation of biological spiking neurons. It models the way the number and strength of excitatory and inhibitory signals entering the neuron affects its spiking frequency BID4.A discrete, as opposed to a continuous-time, network was chosen for a few reasons. First, we wanted to see if the findings of BID12 could be transposed to a discrete-time network, as the researchers themselves where wondering in their paper. This was further motivated by the fact that in the field of artificial intelligence, discretetime networks are widely used BID8 as they are better adapted to the computer architecture. Any link between the study of neural networks for neuroscience and for artificial intelligence would be beneficial to both fields. Last but not least is the fact that discrete-time networks are much easier and faster to implement than their continuous counterparts, and a wide array of resources to train them are freely available. The W in, W and W out weights matrices were originally set randomly (using a linear probability function) between − for W and W out where N is the number of neurons. This strategy is often used in the literature to get an initial output with a probability distribution approaching a normal law with average 0 and variance 1. Only the W and W out matrices were trained. Since the network is recurrent, the output is potentially the of an infinite number of steps, which complicates training. The solution is to "unfold" the network, i.e., to consider only a certain number of steps back in time during training. In figure 5b, we see the unfolded version of the network. By trial and error we chose a 10 time-steps deep unfolded network for training. 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 Investigation of Memory in RNNs There are now slow points with more than three unstable dimensions. Those are marked by red triangles and the number of unstable dimensions is written above. We posit that this is a of the networks' capacity being stretched to the limit (it could not perform the task for longer delays) and not an actual change in strategy for the long-term memory task.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We investigate how a recurrent neural network successfully learns a task combining long-term memory and sequential recall.

Represent the short summary for finding the corresponding scientific article (long)

We propose a continual version of Equilibrium Propagation, where neuron and synapse dynamics occur simultaneously throughout the second phase, with theoretical guarantees and numerical simulations.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The problem of exploration in reinforcement learning is well-understood in the tabular case and many sample-efficient algorithms are known. Nevertheless, it is often unclear how the algorithms in the tabular setting can be extended to tasks with large state-spaces where generalization is required. Recent promising developments generally depend on problem-specific density models or handcrafted features. In this paper we introduce a simple approach for exploration that allows us to develop theoretically justified algorithms in the tabular case but that also give us intuitions for new algorithms applicable to settings where function approximation is required. Our approach and its underlying theory is based on the substochastic successor representation, a concept we develop here. While the traditional successor representation is a representation that defines state generalization by the similarity of successor states, the substochastic successor representation is also able to implicitly count the number of times each state (or feature) has been observed. This extension connects two until now disjoint areas of research. We show in traditional tabular domains (RiverSwim and SixArms) that our algorithm empirically performs as well as other sample-efficient algorithms. We then describe a deep reinforcement learning algorithm inspired by these ideas and show that it matches the performance of recent pseudo-count-based methods in hard exploration Atari 2600 games. Reinforcement learning (RL) tackles sequential decision making problems by formulating them as tasks where an agent must learn how to act optimally through trial and error interactions with the environment. The goal in these problems is to maximize the sum of the numerical reward signal observed at each time step. Because the actions taken by the agent influence not just the immediate reward but also the states and associated rewards in the future, sequential decision making problems require agents to deal with the trade-off between immediate and delayed rewards. Here we focus on the problem of exploration in RL, which aims to reduce the number of samples (i.e., interactions) an agent needs in order to learn to perform well in these tasks when the environment is initially unknown. The sample efficiency of RL algorithms is largely dependent on how agents select exploratory actions. In order to learn the proper balance between immediate and delayed rewards agents need to navigate through the state space to learn about the outcome of different transitions. The number of samples an agent requires is related to how quickly it is able to explore the state-space. Surprisingly, the most common approach is to select exploratory actions uniformly at random, even in high-profile success stories of RL (e.g., BID26 BID17 . Nevertheless, random exploration often fails in environments with sparse rewards, that is, environments where the agent observes a reward signal of value zero for the majority of states. In model-based approaches agents explicitly learn a model of the dynamics of the environment which they use to plan future actions. In this setting the problem of exploration is well understood. When all states can be enumerated and uniquely identified (tabular case), we have algorithms with proven sample complexity bounds on the maximum number of suboptimal actions an agent selects before 1 When we refer to environments with sparse rewards we do so for brevity and ease of presentation. Actually, any sequential decision making problem has dense rewards. In the RL formulation a reward signal is observed at every time step. By environments with sparse rewards we mean environments where the vast majority of transitions lead to reward signals with the same value.converging to an -optimal policy (e.g., BID4 BID10 BID23 . However, these approaches are not easily extended to large environments where it is intractable to enumerate all of the states. When using function approximation, the concept of state visitation is not helpful and learning useful models is by itself quite challenging. Due to the difficulties in learning good models in large domains, model-free methods are much more popular. Instead of building an explicit model of the environment, they estimate state values directly from transition samples (state, action, reward, next state). Unfortunately, this approach makes systematic exploration much more challenging. Nevertheless, because model-free methods make up the majority of approaches scalable to large domains, practitioners often ignore the exploration challenges these methods pose and accept the high sample complexity of random exploration. Reward bonuses that promote exploration are one alternative to random walks (e.g., BID2 BID15, but none such proposed solutions are widely adopted in the field. In this paper we introduce an algorithm for exploration based on the successor representation (SR). The SR, originally introduced by BID5, is a representation that generalizes between states using the similarity between their successors, i.e., the similarity between the states that follow the current state given the environment's dynamics and the agent's policy. The SR is defined for any problem, it can be learned through temporal-difference learning BID25 and, as we discuss below, it can also be seen as implicitly estimating the transition dynamics of the environment. Our approach is inspired by the substochastic successor representation (SSR), a concept we introduce here. The SSR is defined so that it implicitly counts state visitation, allowing us to use it to encourage exploration. This idea connects representation learning and exploration, two otherwise disjoint areas of research. The SSR allows us to derive an exploration bonus that when applied to model-based RL generates algorithms that perform as well as theoretically sample-efficient algorithms. Importantly, the intuition developed with the SSR assists us in the design of a model-free deep RL algorithm that achieves performance similar to pseudo-count-based methods in hard exploration Atari 2600 games BID2 BID20. We consider an agent interacting with its environment in a sequential manner. Starting from a state S 0 ∈ S, at each step the agent takes an action A t ∈ A, to which the environment responds with a state S t+1 ∈ S according to a transition probability function p(s |s, a) = Pr(S t+1 = s |S t = s, A t = a), and with a reward signal R t+1 ∈ R, where r(s, a) indicates the expected reward for a transition from state s under action a, that is, r(s, a). DISPLAYFORM0 The value of a state s when following a policy π, v π (s), is defined to be the expected sum of discounted rewards from that state: DISPLAYFORM1, with γ being the discount factor. When the transition probability function p and the reward function r are known, we can compute v π (s) recursively by solving the system of equations below BID3: DISPLAYFORM2 This equation can also be written in matrix form with v π, r ∈ R |S| and P π ∈ R |S|×|S|: DISPLAYFORM3 where P π is the state to state transition probability function induced by π, that is, P π (s, s) = a π(a|s)p(s |s, a). Traditional model-based algorithms for RL work by learning estimates of the matrix P π and of the vector r and using them to estimate v π, for example by solving Equation 1. We useP π andr to denote empirical estimates of P π and r. Formally, DISPLAYFORM4 wherer(i) denotes the i-th entry in the vectorr, n(s, s) is the number of times the transition s → s was observed, n(s) = s ∈S n(s, s), and C(s, s) is the sum of the rewards associated with the n(s, s) transitions (we drop the action in the discussion to simplify notation).Alternatively, in model-free RL, instead of estimating P π and r we estimate v π (s) directly from samples. We often use temporal-difference (TD) learning BID25 to update our estimates of DISPLAYFORM5 where α is the step-size parameter. Generalization is required in problems with large state spaces, where it is unfeasible to learn an individual value for each state. We do so by parametrizingv(s) with a set of weights θ. We write, given the weights θ,v(s; θ) ≈ v π (s) andq(s, a; θ) ≈ q π (s, a), where q π (s, a) = r(s, a) + γ s p(s |s, a)v π (s). Model-free methods have performed well in problems with large state spaces, mainly due to the use of neural networks as function approximators (e.g., BID17 .Our algorithm is based on the successor representation (SR; BID5 . The successor representation, with respect to a policy π, Ψ π, is defined as DISPLAYFORM6 where we assume the sum is convergent with I denoting the indicator function. BID5 has shown that this expectation can be estimated from samples through TD learning. It also corresponds to the Neumann series of γP : DISPLAYFORM7 Notice that the SR is part of the solution when computing a value function: DISPLAYFORM8 We useΨ π to denote the SR computed throughP π, the approximation of P π .The definition of the SR can also be extended to features. Successor features generalize the SR to the function approximation setting BID0 . We use the definition for the uncontrolled case in this paper. Importantly, the successor features can also be learned with TD learning. Definition 2.1 (Successor Features). For a given 0 ≤ γ < 1, policy π, and for a feature representation φ(s) ∈ R d, the successor features for a state s are: DISPLAYFORM9 Notice that this definition reduces to the SR in the tabular case, where Φ = I. In this section we introduce the concept of the substochastic successor representation (SSR). The SSR is derived from an empirical transition matrix similar to Equation 2, but where each state incorporates a small (1/(n(s) + 1)) probability of terminating at that state, rather than transiting to a next state. As we will show, we can recover the visit counts n(s) through algebraic manipulation on the SSR.While computing the SSR is usually impractical, we use it as inspiration in the design of a new deep reinforcement learning algorithm for exploration (Section 4). In a nutshell, we view the SSR as approximating the process of learning the SR from an uninformative initialization (i.e., the zero vector), and using a stochastic update rule. While this approximation is relatively coarse, we believe it gives qualitative justification to our use of the learned SR to guide exploration. To further this claim, we demonstrate that using the SSR in synthetic, tabular settings yields comparable performance to that of theoretically-derived exploration algorithms. Definition 3.1 (Substochastic Successor Representation). LetP π denote the substochastic matrix induced by the environment's dynamics and by the policy π such thatP π (s |s) = n(s,s) n(s)+1. For a given 0 ≤ γ < 1, the substochastic successor representation,Ψ π, is defined as: DISPLAYFORM0 The theorem below formalizes the idea that the 1 norm of the SSR implicitly counts state visitation. Theorem 1. Let n(s) denote the number of times state s has been visited and let χ(s) = (1 + γ) − ||Ψ π (s)|| 1, whereΨ π is the substochastic SR as in Definition 3.1. For a given 0 ≤ γ < 1, DISPLAYFORM1 Proof of Theorem 1. LetP π be the empirical transition matrix. We first rewriteP π in terms ofP π: DISPLAYFORM2 The expression above can also be written in matrix form:P π = (I − N)P π, where N ∈ R |S|×|S| denotes the diagonal matrix of augmented inverse counts. ExpandingΨ π we have: DISPLAYFORM3 The top eigenvector of a stochastic matrix is the all-ones vector, e , and it corresponds to the eigenvalue 1. Using this fact and the definition ofP π with respect toP π we have: DISPLAYFORM4 We can now bound the term γ π e using the fact that e is also the top eigenvector of the successor representation and has eigenvalue DISPLAYFORM0 Plugging FORMULA14 into the definition of χ we have (notice that Ψ(s)e = ||Ψ(s)|| 1 ): DISPLAYFORM1 When we also use the other bound on the quadratic term we conclude that, for any state s, DISPLAYFORM2 In other words, the SSR, obtained after a slight change to the SR, can be used to recover state visitation counts. The intuition behind this is that the phantom transition, represented by the +1 in the denominator of the SSR, serves as a proxy for the uncertainty about that state by underestimating the SR. This is due to the fact that s P π (s, s) gets closer to 1 each time state s is visited. This can now be used to convert the SSR into a reward function in the tabular case. We do so by using the SSR to define an exploration bonus, r int, such that the reward being maximized by the agent becomes r(s, a) + βr int (s), where β is a scaling parameter. Since we want to incentivize agents to visit the least visited states as quickly as possible, we can trivially define r int = −||Ψ π (s)|| 1, where we penalize the agent by visiting the states that lead to commonly visited states. Notice that the shift (1 + γ) in χ(s) has no effect as an exploration bonus because it is the same across all states. BID23. The performance of our algorithm is the average over 100 runs. A 95% confidence interval is reported between parentheses. E 3 R-MAX MBIE ESSR RIVERSWIM 3,000,000 3,000,000 3,250,000 3,088,924 (± 57,584) SIXARMS 1,800,000 2,800,000 9,250,000 7,327,222 (± 1,189,460) DISPLAYFORM3 We evaluated the effectiveness of the proposed exploration bonus in a standard model-based algorithm. In our implementation the agent updates its transition probability model and reward model through Equation 2 and its SSR estimate as in Definition 3.1 (the pseudo-code of this algorithm is available in the Appendix), which is then used for the exploration bonus r int. We used the domains RiverSwim and SixArms BID23 to assess the performance of this algorithm. 2 These are traditional domains in the PAC-MDP literature BID9 and are often used to evaluate provably sampleefficient algorithms. Details about these environments are also available in the Appendix. We used the same protocol used by BID23. Our are available in TAB0. It is interesting to see that our algorithm performs as well as R-MAX BID4 ) and E 3 on RiverSwim and it clearly outperforms these algorithms on SixArms. In large environments, where enumerating all states is not an option, directly using the SSR as described in the previous section is not viable. Learning the SSR becomes even more challenging when the representation, φ(·), is also being learned and so is non-stationary. In this section we design an algorithm for the function approximation setting inspired by the from the previous section. Since explicitly estimating the transition probability function is not an option, we learn the SR directly using TD learning. In order to capture the SSR we rely on TD's tendency to underestimate values when the estimates are pessimistically initialized, just as the SSR underestimates the true successor representation; with larger underestimates for states (and similarly features) that are rarely observed. This is mainly due to the fact that when the SR is being learned with TD learning, because a reward of 1 is observed at each time step, there is no variance in the target and the predictions slowly approach the true value of the SR. When pessimistically initialized, the predictions approach the target from below. In this sense, what defines how far a prediction is from its final target is indeed how many times it has been updated in a given state. Finally, recent work BID11 BID14 have shown successor features can be learned jointly with the feature representation itself. These ideas are combined together to create our algorithm. The neural network we used to learn the agent's value function while also learning the feature representation and the successor representation is depicted in Figure 1. The layers used to compute the state-action value function,q(S t, ·), are structured as in DQN BID17, but with different numbers of parameters (i..e, filter sizes, stride, and number of nodes). This was done to match Oh et al.'s architecture, which is known to succeed in the auxiliary task we define below. From here on, we will call the part of our architecture that predictsq(S t, ·) DQN e. It is trained to minimize DISPLAYFORM0 DISPLAYFORM1 This loss is known as the mixed Monte-Carlo return (MMC) and it has been used in the past by the algorithms that achieved succesful exploration in deep reinforcement learning BID2 BID20. The distinction between θ and θ − is standard in the field, with θ − denoting the parameters of the target network, which is updated less often for stability purposes BID17. As before, we use r int to denote the exploration bonus obtained from the successor features of the internal representation, φ(·), which will be defined below. Moreover, to ensure all features are in the same range, we normalize the feature vector so that ||φ(·)|| 2 = 1. In Figure 1 we highlight the layer in which we normalize its output with the symbol φ. Notice that the features are always non-negative due to the use of ReLU gates. The successor features are computed by the two bottom layers of the network, which minimize the loss DISPLAYFORM2 Zero is a fixed point for the SR. This is particularly concerning in settings with sparse rewards. The agent might learn to set φ(·) = 0 to achieve zero loss. We address this problem by not propagating ∇L SR to φ(·) (this is depicted in Figure 1 as an open circle stopping the gradient), and by creating an auxiliary task BID7 to encourage a representation to be learned before a non-zero reward is observed. As Machado et al. FORMULA3, we use the auxiliary task of predicting the next observation, learned through the architecture proposed by BID18, which is depicted as the top layers in Figure 1. The loss we minimize for this last part of the network is L Recons = Ŝ t+1 − S t+1 2. The last step in describing our algorithm is to define r int (S t ; θ −), the intrinsic reward we use to encourage exploration. We choose the exploration bonus to be the inverse of the 2 -norm of the vector of successor features of the current state, that is, DISPLAYFORM0 where ψ(S t ; θ −) denotes the successor features of state S t parametrized by θ −. The exploration bonus comes from the same intuition presented in the previous section, but instead of penalizing the agent with the norm of the SR we make r int (S t ; θ −) into a bonus (we observed in preliminary experiments not discussed here that DQN performs better when dealing with positive rewards). Moreover, instead of using the 1 -norm we use the 2 -norm of the SR since our features have unit length in 2 (whereas the successor probabilities in the tabular-case have unit length in 1).Finally, we initialize our network the same way BID18 does. We use Xavier initialization BID6 in all layers except the fully connected layers around the element-wise multiplication denoted by ⊗, which are initialized uniformly with values between −0.1 and 0.1. We followed the evaluation protocol proposed by BID13. We used MONTEZUMA'S REVENGE to tune our parameters (training set). The reported are the average over 10 seeds after 100 million frames. We evaluated our agents in the stochastic setting (sticky actions, ς = 0.25) using a frame skip of 5 with the full action set (|A| = 18). The agent learns from raw pixels, that is, it uses the game screen as input. Our were obtained with the algorithm described in Section 4. We set β = 0.025 after a rough sweep over values in the game MONTEZUMA'S REVENGE. We annealed in DQN's -greedy exploration over the first million steps, starting at 1.0 and stopping at 0.1 as done by BID2. We trained the network with RMSprop with a step-size of 0.00025, an value of 0.01, and a decay of 0.95, which are the standard parameters for training DQN BID17. The discount factor, γ, is set to 0.99 and w TD = 1, w SR = 1000, w Recons = 0.001. The weights w TD, w SR, and w Recons were set so that the loss functions would be roughly the same scale. All other parameters are the same as those used by BID17. TAB2 summarizes the after 100 million frames. The performance of other algorithms is also provided for reference. Notice we are reporting learning performance for all algorithms instead of the maximum scores achieved by the algorithm. We use the superscript MMC to distinguish between the algorithms that use MMC from those that do not. When comparing our algorithm, DQN MMC e +SR, to DQN we can see how much our approach improves over the most traditional baseline. By comparing our algorithm's performance to DQN MMC +CTS BID2 and DQN MMC +PixelCNN BID20 we compare our algorithm to established baselines for exploration. As highlighted in Section 4, the parameters of the network we used are different from those used in the traditional DQN network, so we also compared the performance of our algorithm to the performance of the same network our algorithm uses but without the additional modules (next state prediction and successor representation) by setting w SR = w Recons = 0 and without the intrinsic reward bonus by setting β = 0.0. The column labeled DQN MMC e contains the for this baseline. This comparison allows us to explicitly quantify the improvement provided by the proposed exploration bonus. The learning curves of these algorithms, their performance after different amounts of experience, and additional analyzing, for example, the impact of the introduced auxiliary task, are available in the Appendix. We can clearly see that our algorithm achieves scores much higher than those achieved by DQN, which struggles in games that pose hard exploration problems. Moreover, by comparing DQN MMC e +SR to DQN MMC e we can see that the provided exploration bonus has a big impact in the game MONTEZUMA'S REVENGE, which is probably known as the hardest game among those we used in our evaluation. Interestingly, the change in architecture and the use of MMC leads to a big improvement in games such as GRAVITAR and VENTURE, which we cannot fully explain. However, notice that the change in architecture does not have any effect in MONTEZUMA'S REVENGE. The proposed exploration bonus seems to be essential in this game. Finally, we also compared our algorithm to DQN MMC +CTS and DQN MMC +PixelCNN. We can observe that, on average, it performs as well as these algorithms, but instead of requiring a density model it requires the SR, which is already defined for every problem since it is a component of the value function estimates, as discussed in Section 2. There are multiple algorithms in the tabular, model-based case with guarantees about their performance (e.g., BID4 BID10 BID23 BID19 . RiverSwim and SixArms are domains traditionally used when evaluating these algorithms. In this paper we have given evidence that our algorithm performs as well as some of these algorithms with theoretical guarantees. Among these algorithms, R-MAX seems the closest approach to ours. As with R-MAX, the algorithm we presented in Section 3 augments the state-space with an imaginary state and encourages the agent to visit that state, implicitly reducing the algorithm's uncertainty in the state-space. However, R-MAX deletes the transition to this imaginary state once a state has been visited a given number of times. Ours lets the probability of visiting this imaginary state vanish with additional visitations. Moreover, notice that it is not clear how to apply these traditional algorithms such as R-MAX and E 3 to large domains where function approximation is required. Conversely, there are not many model-free approaches with proven sample-complexity bounds (e.g., BID24, but there are multiple model-free algorithms for exploration that actually work in large domains (e.g., BID22 BID2 BID20 BID21 . Among these algorithms, the use of pseudo-counts through density models is the closest to ours BID2 BID20 . Inspired by those papers we used the mixed Monte-Carlo return as a target in the update rule. In Section 5 we have shown that our algorithm performs generally as well as these approaches without requiring a density model. Importantly, BID15 had already shown that counting activations of fixed, handcrafted features in Atari 2600 games leads to good exploration behavior. Nevertheless, by using the SSR we are not only counting learned features but we are also implicitly capturing the induced transition dynamics. Finally, the SR has already been used in the context of exploration. However, it was used to help the agent learn how to act in a higher level of abstraction in order to navigate through the state space faster BID12 BID14 . Such an approach has led to promising in the tabular case but only anecdotal evidence about its scalability has been provided when the idea was applied to large domains such as Atari 2600 games. Importantly, the work developed by BID14, BID11 and BID18 are the main motivation for the neural network architecture presented here. BID18 have shown how one can predict the next screen given the current observation and action (our auxiliary task), while BID14 and BID11 have proposed different architectures for learning the successor representation from raw pixels. RL algorithms tend to have high sample complexity, which often prevents them from being used in the real-world. Poor exploration strategies is one of the main reasons for this high sample-complexity. Despite all of its shortcomings, uniform random exploration is, to date, the most commonly used approach for exploration. This is mainly due to the fact that most approaches for tackling the exploration problem still rely on domain-specific knowledge (e.g., density models, handcrafted features), or on having an agent learn a perfect model of the environment. In this paper we introduced a general method for exploration in RL that implicitly counts state (or feature) visitation in order to guide the exploration process. It is compatible to representation learning and the idea can also be adapted to be applied to large domains. This opens up multiple possibilities for future work. Based on the presented in Section 3, for example, we conjecture that the substochastic successor representation can be actually used to generate algorithms with PAC-MDP bounds. Investigating to what extent different auxiliary tasks impact the algorithm's performance, and whether simpler tasks such as predicting feature activations or parts of the input BID7 are effective is also worth studying. Finally, it might be interesting to further investigate the connection between representation learning and exploration, since it is also known that better representations can lead to faster exploration BID8. This supplementary material contains details omitted from the main text due to space constraints. The list of contents is below:• Pseudo-code of the model-based algorithm discussed in Section 3;• Description of RiverSwim and SixArms, the tabular domains we used in our evaluation;• Learning curves of DQN e and DQN MMC e +SR and their performance after different amounts of experience in the Atari 2600 games used for evaluation;• Results of additional experiments designed to evaluate the role of the auxiliary task in the reported in the paper for ESSR. In the main paper we described our algorithm as a standard model-based algorithm where the agent updates its transition probability model and reward model through Equation 2 and its SSR estimate as in Definition 3.1. The pseudo-code with details about the implementation is presented in Algorithm 1. n(s, s) ← 0 ∀s, s ∈ S t(s, a, s) ← 1 ∀s, s ∈ S, ∀a ∈ Â r(s, a) ← 0 ∀s ∈ S, ∀a ∈ Â P (s, a) ← 1/|S| ∀s ∈ S, ∀a ∈ Ã P (s, s) ← 0 ∀s, s ∈ S π ← random over A while episode is not over do Observe s ∈ S, take action a ∈ A selected according to π(s), and observe a reward R and a next state s ∈ S n(s, s) ← n(s, s) + 1 DISPLAYFORM0 r int ← −Ψe π ← POLICYITERATION(P,r + βr int) end while The two domains we used as testbed to evaluate the proposed model-based algorithm with the exploration bonus generated by the substochastic successor representation are shown in FIG2. These domains are the same used by BID23. For SixArms, the agent starts in state 0. For RiverSwim, the agent starts in either state 1 or 2 with equal probability. The algorithm we introduced in the paper, ESSR, relies on a network that estimates the state-action value function, the successor representation, and the next observation to be seen given the agent's current observation and action. While the depicted in TAB2 allow us to clearly see the benefit of using an exploration bonus derived from the successor representation, they do not inform us about the impact of the auxiliary task in the . The experiments in this section aim at addressing this issue. We focus on Montezumas Revenge because it is the game where the problem of exploration is maximized, with most algorithms not being able to do anything without an exploration bonus. The first question we asked was whether the auxiliary task was necessary in our algorithm. We evaluated this by dropping the reconstruction module from the network to test whether the initial random noise generated by the successor representation is enough to drive representation learning. It is not. When dropping the auxiliary task, the average performance of this baseline over 4 seeds in MON-TEZUMA'S REVENGE after 100 million frames was 100.0 points (σ 2 = 200.0; min: 0.0, max: 400.0). As comparison, our algorithm obtains 1778.6 points (σ 2 = 903.6, min: 400.0, max: 2500.0). These suggest that auxiliary tasks seem to be necessary for our method to perform well. We also evaluated whether the auxiliary task was sufficient to generate the we observed. To do so we dropped the SR module and set β = 0.0 to evaluate whether our exploration bonus was actually improving the agent's performance or whether the auxiliary task was doing it. The exploration bonus seems to be essential in our algorithm. When dropping the exploration bonus and the successor representation module, the average performance of this baseline over 4 seeds in MONTEZUMA'S REVENGE after 100 million frames was 398.5 points (σ 2 = 230.1; min: 0.0, max: 400.0). Again, clearly, the auxiliary task is not a sufficient condition for the performance we report. The reported use the same parameters as those reported in the main paper. Learning curves for each individual run are depicted in Figure 3. after different amounts of experience (10, 50, and 100 million frames) in TAB5 Finally, Figure 4 depicts the learning curves obtained with the evaluated algorithms in each game. Lighter lines represent individual runs while the solid lines encode the average over the multiple runs. learning curves in the Atari 2600 games used as testbed. The curves are smoothed with a running average computed using a window of size 100. DISPLAYFORM0

Represent the short summary for finding the corresponding scientific article (long):

We propose the idea of using the norm of the successor representation an exploration bonus in reinforcement learning. In hard exploration Atari games, our the deep RL algorithm matches the performance of recent pseudo-count-based methods.

Represent the TLDR for finding the corresponding scientific article

We propose a model-based method called "Search with Amortized Value Estimates" (SAVE) which leverages both real and planned experience by combining Q-learning with Monte-Carlo Tree Search, achieving strong performance with very small search budgets.