Question: Represent the scientific article to find a short TLDR

Question: Combining multiple function approximators in machine learning models typically leads to better performance and robustness compared with a single function. In reinforcement learning, ensemble algorithms such as an averaging method and a majority voting method are not always optimal, because each function can learn fundamentally different optimal trajectories from exploration. In this paper, we propose a Temporal Difference Weighted (TDW) algorithm, an ensemble method that adjusts weights of each contribution based on accumulated temporal difference errors. The advantage of this algorithm is that it improves ensemble performance by reducing weights of Q-functions unfamiliar with current trajectories. We provide experimental for Gridworld tasks and Atari tasks that show significant performance improvements compared with baseline algorithms. Using ensemble methods that combine multiple function approximators can often achieve better performance than a single function by reducing the variance of estimation . Ensemble methods are effective in supervised learning, and also reinforcement learning . There are two situations where multiple function approximators are combined: combining and learning multiple functions during training and combining individually trained functions to jointly decide actions during testing . In this paper, we focus on the second setting of reinforcement learning wherein each function is trained individually and then combined them to achieve better test performance. Though there is a body of research on ensemble algorithms in reinforcement learning, it is not as sizeable as the research devoted to ensemble methods for supervised learning. investigated many ensemble approaches combining several agents with different valuebased algorithms in Gridworld settings. Faußer & Schwenker (2011; 2015a) have shown that combining value functions approximated by neural networks improves performance greater than using a single agent. Although previous work dealt with each agent equally contributing to the final output, weighting each contribution based on its accuracy is also a known and accepted approach in supervised learning . However, unlike supervised learning, reinforcement learning agents learn from trajectories ing from exploration, such that each agent learns from slightly different data. This characteristic is significant in tasks with high-dimensional state-space, where there are several possible optimal trajectories to maximize cumulative rewards. In such a situation, the final joint policy function ing from simple averaging or majority voting is not always optimal if each agent learned different optimal trajectories. Furthermore, it is difficult to decide constant weights of each contribution as it is possible that agents with poor episode rewards have better performance in specific areas. In this paper, we propose the temporal difference weighted (TDW) algorithm, an ensemble method for reinforcement learning at test time. The most important point of this algorithm is that confident agents are prioritized to participate in action selection while contributions of agents unfamiliar with the current trajectory are reduced. To do so in the TDW algorithm, the weights of the contributions at each Q-function are calculated as softmax probabilities based on accumulated TD errors. Extending an averaging method and a majority voting method, actions are determined by weighted average or voting methods according to the weights. The advantage of the TDW algorithm is that arbitrary training algorithms can use this algorithm without any modifications, because the TDW algorithm only cares about the joint decision problem, which could be easily adopted in competitions and development works using reinforcement learning. In our experiment, we demonstrate that the TDW retains performance in tabular representation Gridworld tasks with multiple possible trajectories, where simple ensemble methods are significantly degraded. Second, to demonstrate the effectiveness of our TDW algorithm in high-dimensional state-space, we also show that our TDW algorithm can achieve better performance than baseline algorithms in Atari tasks . Ensemble methods that combine multiple function approximators during training rather than evaluation have been studied in deep reinforcement learning. Bootstrapped deep Q-network (DQN) leverages multiple heads that are randomly initialized to improve exploration, because each head leads to slightly different states. Averaged-DQN reduces the variance of a target approximation by calculating an average value of last several learned Qnetworks. Using multiple value functions to reduce variance of target estimation is also utilized in the policy gradients methods . In contrast, there has been research focused on joint decision making in reinforcement learning. Using multiple agents to jointly select an action achieves better performance than a single agent (; Faußer & Schwenker (2015a;). However, such joint decision making has been limited to relatively small tasks such as Gridworld and Maze. Therefore, it is not known whether joint decision making with deep neural networks can improve performance in high-dimensional state-space tasks such as Atari 2600 . proposes a multiple mode-based reinforcement learning (MMRL), a weighted ensemble method for model-based reinforcement learning, which determines each weight based by using prediction models. The MMRL gives larger weights to reinforcement learning controllers with small errors of special responsibility predictors. Unlike MMRL, our method does not require additional components to calculate weights. Our method is not the first one to use TD errors in combining multiple agents. proposes a module selection mechanism that chooses the module with smallest TD errors to learn current states, which will eventually assign each module to a small area of a large task. As a joint decision making method, a selective ensemble method is proposed to eliminate agents with less confidence at the current state by measuring TD errors (Faußer & Schwenker (2015b) ), which is the closest approach to our method. This selection drops all outputs whose TD errors exceeds a threshold, which can be viewed as a hard version of our method that uses a softmax of all weighted outputs instead of elimination. The threshold is not intuitively determined. Because the range of TD errors varies by tasks and reward settings, setting the threshold requires sensitive tuning. We formulate standard reinforcement learning setting as follows. At time t, an agent receives a state s t ∈ S, and takes an action a t ∈ A based on a policy function a t = π(s t). The next state s t+1 is given to the agent along with a reward r t+1. The return is defined as a discounted cumulative reward, where γ ∈ is a discount factor. The true value of taking an action a t at a state s t is described as follows: where Q π (s t, a t) is an action-value under the policy π. The optimal value is Q * (s t, a t) = max π Q π (s t, a t). With such an optimal Q-function, optimal actions can be determined based on the highest action-values at each state. DQN is a deep reinforcement learning method that approximates an optimal Qfunction with deep neural networks. The Q-function Q(s t, a t |θ) with a parameter θ is approximated by a Q-learning style update . The parameter θ is learned to minimize squared temporal difference errors. where y t = r t+1 + γ max a Q(s t+1, a|θ) with a target network parameter θ. The target network parameter θ is synchronized to the parameter θ in a certain interval. DQN also introduces use of the experience replay , which randomly samples past state transitions from the replay buffer to compute the squared TD error. Assume there are N sets of trained Q-function Q(s, a|θ i) where i denotes an index of the function. The final policy π(s t) is determined by combining the N Q-functions. We formulate two baseline methods commonly used in ensemble algorithms: Average policy and Majority Voting (MV) policy (Faußer & Schwenker (2011; 2015a); ). Majority Voting (MV) policy is an approach to decide the action based on greedy selection according to the formula: where v i (s, a) is a binary function that outputs 1 for the most valued action and 0 for others: Contributions of each function to the final output are completely equal. Average policy is a method that averages all the outputs of the Q-functions, and the action is greedily determined: Averaging outputs from multiple approximated functions reduces variance of prediction. Unlike MV policy, Average policy leverages all estimated values as well as the highest values. In this section, we explain the TDW ensemble algorithm that adjusts weights of contributions based on accumulated TD errors. The TDW algorithm is especially powerful in the complex situation such as high-dimensional state-space where it is difficult to cover whole state-space with a single agent. Section 4.1 describes the error accumulation mechanism. Section 4.2 introduces joint action selection using the weights computed with the accumulated errors. We consider that a squared TD error δ 2 fundamentally consists of two kinds of errors: where δ p is a prediction error of approximated function, and δ u is an error at states where the agent rarely experienced. In a tabular-based value function, δ p will be near 0 at frequently visited states. In contrast, δ u will be extremely large at less visited states with both a tabular-based value function and a function approximator because TD errors are not sufficiently propagated such a state. There are two causes of unfamiliar states: states are difficult to visit due to hard exploration, and states are not optimal to the agent according to learned state transitions. For combining multiple agents at a joint decision, the second case is noteworthy because each agent may be optimized at different optimal trajectories. Thus, some of the agents will produce larger δ u when they face such states as a of an ensemble, and contributions of less confident agents can be reduced based on the TD error δ u. or. end for To measure uncertainty of less confident agents, we define u i t as a uncertainty of an agent: where α ∈ is a constant factor decaying the uncertainty at a previous step. With a large α, the uncertainty u i is extremely large during unfamiliar trajectories, which makes it possible to easily distinguish confident agents from the others. However, a trade-off arises when prediction error δ p is accumulated for a long horizon, which increases correlation between agents. To reduce contributions of less confident agents, each contribution at joint decision is weighted based on uncertainty u i t. Using the uncertainty u i t, a weight w i t of each agent is calculated as a probability by the softmax function: When the agent has a small uncertainty value u i t, the weight w i t becomes large. We consider two weighted ensemble methods corresponding to the Average policy and the MV policy based on the weights w i t. As a counterpart of the Average policy, our TDW Average policy is as follows: For the MV policy, TDW Voting policy is as follows: Unlike the averaging method, because TDW Voting policy directly uses probabilities calculated by, the correlation between agents can be increased significantly with large decay factor α, leading to worse performance. Although these weighted ensemble algorithms are simple enough to extend to arbitrary ensemble methods, we leave more advanced applications for future work so that we may demonstrate the effectiveness of our approach in a simpler setting. The complete TDW ensemble algorithm is described in Algorithm 1. In this section, we describe the experiments performed on the Gridworld tasks and Atari tasks in Section 5.2. To build the trained Q-functions, we used the table-based Qlearning algorithm and DQN with a standard model, respectively. In each experiment, we evaluated our algorithm to address performance improvements from there baselines as well as the effects of selecting the decay factor α. We first evaluated the TDW algorithms with a tabular representation scenario to show their effectiveness in the situation where it is difficult to cover a whole state-space with the single agent. We built two Gridworld environments as shown in Figure 1. Each environment is designed to induce bias of learned trajectories by setting multiple slits. As a of exploration, once an agent gets through one of the slits to the goal, the agent is easily biased to aim for the same slit due to the max operator of Q-learning. The state-representation is a discrete index of a table with size of 13 × 13. There are four actions corresponding to steps of up, down, left and right. If a wall exists where the agent tries to move, the next state remains the same as the current state. The agent always starts from S depicted in Figure 1. At every timestep, the agent receives a reward of −0.1 or +100 at goal states. The agent starts a new episode if either the agent arrives at the goal states or the timestep reaches 100 steps. We trained N = 10 agents with different random seeds for -greedy exploration with = 0.3. Each training continues until 1M steps have been simulated. We set the learning rate to 0.01 and γ = 0.95. After training, we evaluated TDW ensemble algorithms for 20K episodes. As baselines, we also evaluate each single agent as well as ensemble methods of Average policy and MV policy for 20K episodes each. The evaluation on the Gridworld environments are shown in Table 1. Four-slit Gridworld is significantly more difficult than Two-slit Gridworld because each Q-function is not only horizontally biased, but also vertically biased. In both the Two-slit Gridworld and Four-slit Gridworld environments, the TDW ensemble methods achieve better performance than their corresponding Average policy and the MV policy baselines. Additionally, the of both of the Average policy and the MV policy were worse than the single models. It should be noted that Average policy degrades original performance more than MV policy. For the selection of the decay factor α, a larger α tends to increase performance in TDW Average policy. In contrast, the larger α leads to poor performance in TDW Voting policy especially in Fourslit Gridworld. We believe that the large α significantly reduces contributions of most Q-functions, which would ignore votes of actions that would be the best in equal voting. In contrast, TDW Average policy leverages values of all actions, exploiting all contributions to select the best action. To demonstrate effectiveness in high-dimensional state-space, we evaluated TDW algorithm in Atari tasks. We trained DQN agents across 6 Atari tasks (Asterix, Beamrider, Breakout, Enduro, MsPacman and SpaceInvaders) through OpenAI Gym . At each task, N = 10 agents were trained with different random seeds for neural network initialization, exploration and environments in order to vary the learned Q-function. The training continued until 10M steps (40M game frames) with frame skipping and termination on loss of life enabled. The of exploration is linearly decayed from 1.0 to 0.1 through 1M steps. The hyperparameters of neural networks are same as . After training, evaluation was conducted with each Q-function, TDW Average policy, TDW Voting policy and the two baselines. We additionally evaluated weighted versions of the baselines whose Q-functions were weighted based on their evaluation performance. The evaluation continued for 1000 episodes with = 0.05. The experimental are shown in Table 2. Interestingly, both of Average policy and MV policy improved performance from mean performance of single agents, though the simple ensemble algorithms had not been investigated well in the domain of deep reinforcement learning. In the games of Asterix, Beamrider, Breakout and Enduro, the TDW algorithms achieve additional performance improvements as compared with the non-weighted and weighted baselines. Even in MsPacman and SpaceInvaders, the TDW algorithms perform significantly better than non-weighted baselines and the best single models. In most of the cases, the globally weighted ensemble baselines performed worse than non-weighted versions. We believe this is because these globally weighted ensemble methods will ignore local performance, which is significant in high-dimensional state-space because it is difficult to cover all possible states with single models. The TDW algorithms with small α tend to achieve better performance than those with a large α, which suggests that a significantly large α can increase correlation between Q-functions and reduce contributions of less confident Q-functions. To analyze changes of weights through an episode, we plot entropies during a sample episode on Breakout (TDW Average policy, α = 0.8) in Figure 2 (a). If an entropy is low (high in negative scale), some of Q-functions have large weights, while others have extremely small weights. Extreme low entropies are observed when the ball comes closely to the pad as shown in Figure 2 (b) where the value should be close to 0 for the non-optimal actions because missing the ball immediately ends its life. It is easy for sufficiently learned Q-functions to estimate such a terminal state so that the entropy becomes low due to the gap between optimal Q-functions for the current states and the others. The entropies tend to be low during latter steps as there are many variations of the remaining blocks. We consider that the reason why the TDW algorithms with α = 0.8 achieved the best performance in Breakout is that the large α value reduces influence of the Q-functions which cannot correctly predict values with unseen variations of remaining blocks. In contrast to Breakout where living long leads to higher scores, in SpaceInvaders we observe that the low entropies appear at dodging beams rather than shooting invaders, because shooting beams requires long-term value prediction which does not induce large TD errors. Therefore, performance improvements on SpaceInvaders are not significantly better than weighted baselines. To analyze correlation between the decay factor α and entropies, plots of the number of observations with a certain entropy are shown in Figure 3. In most games, higher decay factors increase the presence of low entropy states and decreases the presence of high entropy states. In the games with frequent reward occurences such as Enduro and MsPacman, there are more low-entropy observations than BeamRider and SpaceInvaders, where reward occurences are less frequent. Especially with regards to MsPacman, we believe that the TDW Average policy with larger α values in worse performance because the agent frequently receives positive rewards at almost every timestep, which often induces prediction error δ p, and increases uncertainty in all Q-functions. Thus, globally weighted ensemble methods achieve better performance than TDW algorithms because it is difficult to consistently accumulate uncertainties on MsPacman. In this paper, we have introduced the TDW algorithm: an ensemble method that accumulates temporal difference errors as an uncertainties in order to adjust weights of each Q-function, improving performance especially in high-dimensional state-space or situations where there are multiple optimal trajectories. We have shown performance evaluations in Gridworld tasks and Atari tasks, wherein the TDW algorithms have achieved significantly better performance than non-weighted algorithms and globally weighted algorithms. However, it is difficult to correctly measure uncertainties with frequent reward occurrences because the intrinsic prediction errors are also accumulated. Thus, these types of games did not realize the same performance improvements. In future work, we intend to investigate an extension of this work into continuous action-space tasks because only the joint decision problem of Q-functions is considered in this paper. We believe a similar algorithm can extend a conventional ensemble method of Deep Deterministic Policy Gradients by measuring uncertainties of pairs of a policy function and a Q-function. We will also consider a separate path, developing an algorithm that measures uncertainties without rewards because reward information is not always available especially in the case of real world application. A Q-FUNCTION TABLES OBTAINED ON GRIDWORLDS table 1 0 20 40 60 80 100 table 2 0 20 40 60 80 100 table 3 0 20 40 60 80 100 table 4 0 20 40 60 80 100 table 5 0

represent this text to find a much much longer article about it with say hundreds-thousands of words
For example, We learn to conditionally generate protein sequences given structures with a model that captures sparse, long-range dependencies. should be similar to Engineered proteins offer the potential to solve many problems in biomedicine, energy, and materials science, but creating designs that succeed is difficult in practice. A significant aspect of this challenge is the complex coupling between protein sequence and 3D structure, and the task of finding a viable design is often referred to as the inverse protein folding problem. We develop generative models for protein sequences conditioned on a graph-structured specification of the design target. Our approach efficiently captures the complex dependencies in proteins by focusing on those that are long-range in sequence but local in 3D space. Our framework significantly improves upon prior parametric models of protein sequences given structure, and takes a step toward rapid and targeted biomolecular design with the aid of deep generative models. A central goal for computational protein design is to automate the invention of protein molecules with defined structural and functional properties. This field has seen tremendous progess in the past two decades BID14, including the design of novel 3D folds BID20, enzymes BID30, and complexes BID4. However, the current practice often requires multiple rounds of trial-and-error, with first designs frequently failing BID19 BID28. Several of the challenges stem from the bottom-up nature of contemporary approaches that rely on both the accuracy of energy functions to describe protein physics as well as on the efficiency of sampling algorithms to explore the protein sequence and structure space. Here, we explore an alternative, top-down framework for protein design that directly learns a conditional generative model for protein sequences given a specification of the target structure, which is represented as a graph over the sequence elements. Specifically, we augment the autoregressive self-attention of recent sequence models BID34 with graph-based descriptions of the 3D structure. By composing multiple layers of structured self-attention, our model can effectively capture higher-order, interaction-based dependencies between sequence and structure, in contrast to previous parameteric approaches BID24 BID36 that are limited to only the first-order effects. The graph-structured conditioning of a sequence model affords several benefits, including favorable computational efficiency, inductive bias, and representational flexibility. We accomplish the first two by leveraging a well-evidenced finding in protein science, namely that long-range dependencies in sequence are generally short-range in 3D space BID23 BID3. By making the graph and self-attention similarly sparse and localized in 3D space, we achieve computational scaling that is linear in sequence length. Additionally, graph structured inputs offer representational flexibility, as they accomodate both coarse,'flexible backbone' (connectivity and topology) as well as fine-grained (precise atom locations) descriptions of structure. We demonstrate the merits of our approach via a detailed empirical study. Specifically, we evaluate our model at structural generalization to sequences of protein folds that were outside of the training set. Our model achieves considerably improved generalization performance over the recent deep models of protein sequence given structure as well as structure-naïve language models. Generative models for proteins A number of works have explored the use of generative models for protein engineering and design. and BID36 proposed neural models for sequences given 3D structure, where the amino acids at different positions in the sequence are predicted independently of one another. BID11 introduced a generative model for protein sequences conditioned on a 1D, context-free grammar based specification of the fold topology. BID8 and BID37 used deep neural networks to model the conditional distribution of letters in a specific position given the structure and sequence of all surrounding residues. In contrast to these works, our model captures the joint distribution of the full protein sequence while grounding these dependencies in terms of long-range interactions arising from the structure. In parallel to the development of structure-based models, there has been considerable work on deep generative models for protein sequences in individual protein families with directed BID26 BID31 and undirected BID33 ) latent variable models. These methods have proven useful for protein engineering, but presume the availability of a large number of sequences from a particular family. More recently, several groups have obtained promising using unconditional protein language models BID6 BID0 BID12 BID27 to learn protein sequence representations that can transfer well to supervised tasks. While serving different purposes, we emphasize that one advantage of conditional generative modeling is to facilitate adaptation to specific (and potentially novel) parts of structure space. Language models trained on hundreds of millions of evolutionary sequences are unfortunately still'semantically' bottlenecked by the much smaller number of evolutionary 3D folds (perhaps thousands) that the sequences design. We propose evaluating protein language models with structure-based splitting of sequence data (Section 3, albeit on much smaller sequence data), and begin to see how unconditional language models may struggle to assign high likelihoods to sequences from out-of-training folds. In a complementary line of research, deep models of protein structure BID2 BID16 BID1 have been proposed recently that could be used to craft 3D structures for input to sequence design. Protein design For classical approaches to computational protein design, which are based on joint modeling of structure and sequence, we refer the reader to a review of both methods and accomplishments in BID14. More recently, proposed a non-parametric approach to protein design in which a target design is decomposed into substructural motifs that are then queried against a protein database. In this work we will focus on comparisons with direct parametric models of the sequence-structure relationship. Self-Attention Our model extends the Transformer BID7 to additionally capture sparse, pairwise relational information between sequence elements. The dense variation of this problem was explored in BID29 and. As noted in those works, incorporating general pairwise information incurs O(N 2) memory (and computational) cost for sequences of length N, which can be highly limiting for training on GPUs. We circumvent this cost by instead restricting the self-attention to the sparsity of the input graph. Given this graph-structured self-attention, our model may also be reasonably cast in the framework of message-passing or graph neural networks BID10 BID5. Our approach is similar to Graph Attention Networks BID35, but augmented with edge features and an autoregressive decoder. We represent protein structure in terms of an attributed graph G = (V, E) with node features V = {v 1, . . ., v N} and edge features E = {e ij} i =j over the sequence residues (amino acids). This formulation can accommodate different variations on the macromolecular design problem, including 3D considerations For a rigid-body design problem, the structure for conditioning is a fixed set of backbone coordinates X = {x i ∈ R 3 : 1 ≤ i ≤ N}, where N is the number of positions 1. We desire a graph representation of the coordinates G(X) that has two properties:• Invariance. The features are invariant to rotations and translations.• Locally informative. The edge features incident to v i due to its neighbors N(i),i.e. {e ij} j∈N(i), contain sufficient information to reconstruct all adjacent coordinates {x j} j∈N(i) up to rigid-body motion. While invariance is motivated by standard symmetry considerations, the second property is motivated by limitations of current graph neural networks BID10. In these networks, updates to node features v i depend only on the edge and node features adjacent to v i. However, typically, these features are insufficient to reconstruct the relative neighborhood positions {x j} j∈N(i), so individual updates cannot fully depend on the'local environment'. For example, pairwise distances D ij and D il are insufficient to determine if x j and x l are on the same or opposite sides of x i. We develop invariant and locally informative features by first augmenting the points x i with'orientations' O i that define a local coordinate system at each point. We define these in terms of the backbone geometry as DISPLAYFORM0 where b i is the negative bisector of angle between the rays (x i−1 − x i) and (x i+1 − x i), and n i is a unit vector normal to that plane. Formally, we have DISPLAYFORM1 Finally, we derive the spatial edge features e (s) ij from the rigid body transformation that relates reference frame (x i, O i) to reference frame (x j, O j). While this transformation has 6 degrees of freedom, we decompose it into features for distance, direction, and orientation as DISPLAYFORM2 Here r(·) is a function that lifts the distances into a radial basis 2, the term in the middle corresponds to the relative direction of x j in the reference frame of (x i, O i), and q(·) converts the 3 × 3 relative rotation matrix to a quaternion representation. Quaternions represent rotations as four-element vectors that can be efficiently and reasonably compared by inner products. Positional encodings Taking a cue from the original Transformer model, we obtain positional embeddings e (p) ij that encode the role of local structure around node i. Specifically, we need to model the positioning of each neighbor j relative to the node under consideration i. Therefore, we obtain the position embedding as a sinusoidal function of the gap i − j. Note that this is in contrast to the absolute positional encodings of the original Transformer, and instead matches the relative encodings in BID29.Node and edge features Finally, we obtain an aggregate edge encoding vector e ij by concatenating the structural encodings e (s) ij and the positional encodings e (p) ij and then linearly transforming them to have the same dimension as the model. We only include edges in the k-nearest neighbors graph of X, with k = 30 for all experiments. For node features, we compute the three dihedral angles of the protein backbone (φ i, ψ i, ω i) and embed these on the 3-torus as {sin, cos} × (φ i, ψ i, ω i). We also consider'flexible backbone' descriptions of 3D structure based solely on topological binary edge features. We combine the relative positional encodings with two binary edge features: contacts that indicate when the distance between C α residues at i and j are less than 8 Angstroms and hydrogen bonds which are directed and defined by the electrostatic model of DSSP BID17. These features implicitly integrate over different 3D backbone configurations that are compatible with the specified topology. In this work, we introduce a Structured Transformer model that draws inspiration from the selfattention based Transformer model BID34 and is augmented for scalable incorporation of relational information. While general relational attention incurs quadratic memory and computation costs, we avert these by restricting the attention for each node i to the set N(i, k) of its k-nearest neighbors in 3D space. Since our architecture is multilayered, iterated local attention can derive progressively more global estimates of context for each node i. Second, unlike the standard Transformer, we also include edge features to embed the spatial and positional dependencies in deriving the attention. Thus, our model generalizes Transformer to spatially structured settings. Autoregressive decomposition We decompose the joint distribution of the sequence given structure p(s|x) autoregressively as DISPLAYFORM0 where the conditional probability p(s i |x, s <i) of amino acid s i at position i is conditioned on both the input structure x and the preceding amino acids s <i = {s 1, . . . s i−1} 4. These conditionals are parameterized in terms of two sub-networks: an encoder that computes refined node embeddings from structure-based node features V(x) and edge features E(x) and a decoder that autoregressively predicts letter s i given the preceding sequence and structural embeddings from the encoder. Encoder Our encoder module is designed as follows. A transformation W h: DISPLAYFORM1 Each layer of the encoder implements a multi-head self-attention component, where head ∈ [L] can attend to a separate subspace of the embeddings via learned query, key and value transformations BID34. The queries are derived from the current embedding at node i while the keys and values from the relational information r ij = (h j, e ij) at adjacent nodes j ∈ N (i, k). Specifically, W DISPLAYFORM2 where m DISPLAYFORM3 The of each attention head l are collected as the weighted sum h DISPLAYFORM4 ij and then concatenated and transformed to give the update DISPLAYFORM5 We update the embeddings with this residual and alternate between these self-attention layers and position-wise feedforward layers as in the original Transformer BID34. We stack multiple layers atop each other, and thereby obtain continually refined embeddings as we traverse the layers bottom up. The encoder yields the embeddings produced by the topmost layer as its output. Decoder Our decoder module has the same structure as the encoder but with augmented relational information r ij that allows access to the preceding sequence elements s <i in a causally consistent manner. Whereas the keys and values of the encoder are based on the relational information r ij = (h j, e ij), the decoder can additionally access sequence elements s j as DISPLAYFORM6 Here h DISPLAYFORM7 is the embedding of node j in the current layer of the decoder, h (enc) j is the embedding of node j in the final layer of the encoder, and g(s j) is a sequence embedding of amino acid s j at node j. This concatenation and masking structure ensures that sequence information only flows to position i from positions j < i, but still allows position i to attend to subsequent structural information. We stack three layers of self-attention and position-wise feedforward modules for the encoder and decoder with a hidden dimension of 128 throughout the experiments 5. Dataset To evaluate the ability of the models to generalize across different protein folds, we collected a dataset based on the CATH hierarchical classification of protein structure BID25. For all domains in the CATH 4.2 40% non-redundant set of proteins, we obtained full chains up to length 500 (which may contain more than one domain) and then cluster-split these at the CATH topology level (i.e. fold level) into training, validation, and test sets at an 80/10/10 split. Chains containing multiple CATH tpologies were purged with precedence for test over validation over train. Our splitting procedure ensured that no two domains from different sets would share the same topologies (folds). The final splits contained 18025 chains in the training set, 1637 chains in the validation set, and 1911 chains in the test set. Optimization We trained models using the learning rate schedule and initialization of BID34, a dropout BID32 rate of 10%, and early stopping based on validation perplexity.5 except for the decoder-only language model experiment which used a hidden dimension of 256 Many protein sequences may reasonably design the same 3D structure BID21, and so we focus on likelihood-based evaluations of model performance. Specifically, we evaluate the perplexity per letter of test protein folds (topologies) that were held out from the training and validation sets. Protein perplexities What kind of perplexities might be useful? To provide context, we first present perplexities for some simple models of protein sequences in TAB0. The amino acid alphabet and its natural frequencies upper-bound perplexity at 20 and ∼17.8, respectively. Random protein sequences under these null models are unlikely to be functional without further selection BID18. First order profiles of protein sequences such as those from the Pfam database BID9, however, are widely used for protein engineering. We found the average perplexity per letter of profiles in Pfam 32 (ignoring alignment uncertainty) to be ∼11.6. This suggests that even models with high perplexities of this order have the potential to be useful models for the space of functional protein sequences. The importance of structure We found that there was a significant gap between unconditional language models of protein sequences and models conditioned on structure. Remarkably, for a range of structure-independent language models, the typical test perplexities turned out to be ∼16-17 TAB1, which were barely better than null letter frequencies TAB0. We emphasize that the RNNs were not broken and could still learn the training set in these capacity ranges. It would seem that protein language models trained on one subset of 3D folds (in our cluster-splitting procedure) generalize poorly to predict the sequences of unseen folds, which is important to consider when training protein language models for protein engineering and design. All structure-based models had (unsurprisingly) considerably lower perplexities. In particular, our Structured Transformer model attained a perplexity of ∼7 on the full test set. When we compared different graph features of protein structure TAB2, we indeed found that using local orientation information was important. We also compared to a recent method SPIN2 that predicts, using deep neural networks, protein sequence profiles given protein structures BID24. Since SPIN2 is computationally intensive (minutes per protein for small proteins) and was trained on complete proteins rather than chains, we evaluated it on two subsets of the full test set: a'Small' subset of the test set containing chains up to length 100 and a'Single chain' subset containing only those models where the single chain accounted for the entire protein record in the Protein Data Bank. Both subsets discarded any chains with structural gaps. We found that our Structured Transformer model considerably improved upon the perplexities of SPIN2 TAB1. We presented a new deep generative model to'design' protein sequences given a graph specification of their structure. Our model augments the traditional sequence-level self-attention of Transformers BID34 with relational 3D structural encodings and is able to leverage the spatial locality of dependencies in molecular structures for efficient computation. When evaluated on unseen folds, the model achieves significantly improved perplexities over the state-of-the-art parametric generative models. Our framework suggests the possibility of being able to efficiently design and engineer protein sequences with structurally-guided deep generative models, and underscores the central role of modeling sparse long-range dependencies in biological sequences. We thank members of the MIT MLPDS consortium for helpful feedback and discussions.

Ensemble method for reinforcement learning that weights Q-functions based on accumulated TD errors.

Represent the short summary for finding the corresponding scientific article (long)

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.


Question: Represent the article for finding a single-sentence TLDR.

Question: This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source this http URL, which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research on the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAI Baselines, Dopamine as well as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent researchers. The reinforcement learning (RL) problem describes an agent interacting with an environment with the goal of maximizing cumulative reward through time . Unlike other branches of control, the dynamics of the environment are not fully known to the agent, but can be learned through experience. Unlike other branches of statistics and machine learning, an RL agent must consider the effects of its actions upon future experience. An efficient RL agent must address three challenges simultaneously: 1. Generalization: be able to learn efficiently from data it collects. 2. Exploration: prioritize the right experience to learn from. 3. Long-term consequences: consider effects beyond a single timestep. The great promise of reinforcement learning are agents that can learn to solve a wide range of important problems. According to some definitions, an agent that can perform at or above human level across a wide variety of tasks is an artificial general intelligence (AGI) . Interest in artificial intelligence has undergone a resurgence in recent years. Part of this interest is driven by the constant stream of innovation and success on high profile challenges previously deemed impossible for computer systems. Improvements in image recognition are a clear example of these accomplishments, progressing from individual digit recognition , to mastering ImageNet in only a few years . The advances in RL systems have been similarly impressive: from checkers (Samuel), to Backgammon , to Atari games (a), to competing with professional players at DOTA or StarCraft and beating the world champions at Go. Outside of playing games, decision systems are increasingly guided by AI systems . As we look towards the next great challenges for RL and AI, we need to understand our systems better . This includes the scalability of our RL algorithms, the environments where we expect them to perform well, and the key issues outstanding in the design of a general intelligence system. We have the existence proof that a single self-learning RL agent can master the game of Go purely from self-play . We do not have a clear picture of whether such a learning algorithm will perform well at driving a car, or managing a power plant. If we want to take the next leaps forward, we need to continue to enhance our understanding. The practical success of RL algorithms has built upon a base of theory including gradient descent , temporal difference learning and other foundational algorithms. Good theory provides insight into our algorithms beyond the particular, and a route towards general improvements beyond ad-hoc tinkering. As the psychologist Kurt Lewin said,'there is nothing as practical as good theory' . If we hope to use RL to tackle important problems, then we must continue to solidify these foundations. This need is particularly clear for RL with nonlinear function approximation, or'deep RL'. At the same time, theory often lags practice, particularly in difficult problems. We should not avoid practical progress that can be made before we reach a full theoretical understanding. The successful development of algorithms and theory typically moves in tandem, with each side enriched by the insights of the other. The evolution of neural network research, or deep learning, provides a poignant illustration of how theory and practice can develop together . Many of the key ideas for deep learning have been around, and with successful demonstrations, for many years before the modern deep learning explosion (; ;). However, most of these techniques remained outside the scope of developed learning theory, partly due to their complex and non-convex loss functions. Much of the field turned away from these techniques in a'neural network winter', focusing instead of function approximation under convex loss . These convex methods were almost completely dominant until the emergence of benchmark problems, mostly for image recognition, where deep learning methods were able to clearly and objectively demonstrate their superiority . It is only now, several years after these high profile successes, that learning theory has begun to turn its attention back to deep learning (; ;). We should not turn away from deep RL just because our current theory is not yet developed. In this paper we introduce the Behaviour Suite for Reinforcement Learning (or bsuite for short): a collection of experiments designed to highlight key aspects of agent scalability. Our aim is that these experiments can help provide a bridge between theory and practice, with benefits to both sides. These experiments embody fundamental issues, such as'exploration' or'memory' in a way that can be easily tested and iterated. For the development of theory, they force us to instantiate measurable and falsifiable hypotheses that we might later formalize into provable guarantees. While a full theory of RL may remain out of reach, the development of clear experiments that instantiate outstanding challenges for the field is a powerful driver for progress. We provide a description of the current suite of experiments and the key issues they identify in Section 2. Our work on bsuite is part of a research process, rather than a final offering. We do not claim to capture all, or even most, of the important issues in RL. Instead, we hope to provide a simple library that collects the best available experiments, and makes them easily accessible to the community. As part of an ongoing commitment, we are forming a bsuite committee that will periodically review the experiments included in the official bsuite release. We provide more details on what makes an'excellent' experiment in Section 2, and on how to engage in their construction for future iterations in Section 5. The Behaviour Suite for Reinforcement Learning is a not a replacement for'grand challenge' undertakings in artificial intelligence, or a leaderboard to climb. Instead it is a collection of diagnostic experiments designed to provide insight into key aspects of agent behaviour. Just as the MNIST dataset offers a clean, sanitised, test of image recognition as a stepping stone to advanced computer vision; so too bsuite aims to instantiate targeted experiments for the development of key RL capabilities. The successful use of illustrative benchmark problems is not unique to machine learning, and our work is similar in spirit to the Mixed Integer Programming Library (MIPLIB) (miplib2017). In mixed integer programming, and unlike linear programming, the majority of algorithmic advances have (so far) eluded theoretical analysis. In this field, MIPLIB serves to instantiate key properties of problems (or types of problems), and evaluation on MIPLIB is a typical component of any new algorithm. We hope that bsuite can grow to perform a similar role in RL research, at least for those parts that continue to elude a unified theory of artificial intelligence. We provide guidelines for how researchers can use bsuite effectively in Section 3. As part of this project we open source github.com/anon/bsuite, which instantiates all experiments in code and automates the evaluation and analysis of any RL agent on bsuite. This library serves to facilitate reproducible and accessible research on the core issues in reinforcement learning. It includes: • Canonical implementations of all experiments, as described in Section 2. • Reference implementations of several reinforcement learning algorithms. • Example usage of bsuite with alternative codebases, including'OpenAI Gym'. • Launch scripts for Google cloud that automate large scale compute at low cost. • A ready-made bsuite Jupyter notebook with analyses for all experiments. • Automated L A T E X appendix, suitable for inclusion in conference submission. We provide more details on code and usage in Section 4. We hope the Behaviour Suite for Reinforcement Learning, and its open source code, will provide significant value to the RL research community, and help to make key conceptual issues concrete and precise. bsuite can highlight bottlenecks in general algorithms that are not amenable to hacks, and reveal properties and scalings of algorithms outside the scope of current analytical techniques. We believe this offers an avenue towards great leaps on key issues, separate to the challenges of large-scale engineering . Further, bsuite facilitates clear, targeted and unified experiments across different code frameworks, something that can help to remedy issues of reproducibility in RL research . The Behaviour Suite for Reinforcement Learning fits into a long history of RL benchmarks. From the beginning, research into general learning algorithms has been grounded by the performance on specific environments . At first, these environments were typically motivated by small MDPs that instantiate the general learning problem.'CartPole' and'MountainCar' are examples of classic benchmarks that has provided a testing ground for RL algorithm development. Similarly, when studying specific capabilities of learning algorithms, it has often been helpful to design diagnostic environments with that capability in mind. Examples of this include'RiverSwim' for exploration or'Taxi' for temporal abstraction . Performance in these environments provide a targeted signal for particular aspects of algorithm development. As the capabilities or RL algorithms have advanced, so has the complexity of the benchmark problems. The Arcade Learning Environment (ALE) has been instrumental in driving progress in deep RL through surfacing dozens of Atari 2600 games as learning environments . Similar projects have been crucial to progress in continuous control , model-based RL and even rich 3D games . Performing well in these complex environments requires the integration of many core agent capabilities. We might think of these benchmarks as natural successors to'CartPole' or'MountainCar'. The Behaviour Suite for Reinforcement Learning offers a complementary approach to existing benchmarks in RL, with several novel components: 1. bsuite experiments enforce a specific methodology for agent evaluation beyond just the environment definition. This is crucial for scientific comparisons and something that has become a major problem for many benchmark suites (Section 2). 2. bsuite aims to isolate core capabilities with targeted'unit tests', rather than integrate general learning ability. Other benchmarks evolve by increasing complexity, bsuite aims to remove all confounds from the core agent capabilities of interest (Section 3). 3. bsuite experiments are designed with an emphasis on scalability rather than final performance. Previous'unit tests' (such as 'Taxi' or 'RiverSwim') are of fixed size, bsuite experiments are specifically designed to vary the complexity smoothly (Section 2). 4. github.com/anon/bsuite has an extraordinary emphasis on the ease of use, and compatibility with RL agents not specifically designed for bsuite. Evaluating an agent on bsuite is practical even for agents designed for a different benchmark (Section 4). This section outlines the experiments included in the Behaviour Suite for Reinforcement Learning 2019 release. In the context of bsuite, an experiment consists of three parts: 1. Environments: a fixed set of environments determined by some parameters. 2. Interaction: a fixed regime of agent/environment interaction (e.g. 100 episodes). 3. Analysis: a fixed procedure that maps agent behaviour to and plots. One crucial part of each bsuite analysis defines a'score' that maps agent performance on the task to. This score allows for agent comparison'at a glance', the Jupyter notebook includes further detailed analysis for each experiment. All experiments in bsuite only measure behavioural aspects of RL agents. This means that they only measure properties that can be observed in the environment, and are not internal to the agent. It is this choice that allows bsuite to easily generate and compare across different algorithms and codebases. Researchers may still find it useful to investigate internal aspects of their agents on bsuite environments, but it is not part of the standard analysis. Every current and future bsuite experiment should target some key issue in RL. We aim for simple behavioural experiments, where agents that implement some concept well score better than those that don't. For an experiment to be included in bsuite it should embody five key qualities: • Targeted: performance in this task corresponds to a key issue in RL. • Simple: strips away confounding/confusing factors in research. • Challenging: pushes agents beyond the normal range. • Scalable: provides insight on scalability, not performance on one environment. • Fast: iteration from launch to in under 30min on standard CPU. Where our current experiments fall short, we see this as an opportunity to improve the Behaviour Suite for Reinforcement Learning in future iterations. We can do this both through replacing experiments with improved variants, and through broadening the scope of issues that we consider. We maintain the full description of each of our experiments through the code and accompanying documentation at github.com/anon/bsuite. In the following subsections, we pick two bsuite experiments to review in detail:'memory length' and'deep sea', and review these examples in detail. By presenting these experiments as examples, we can emphasize what we think makes bsuite a valuable tool for investigating core RL issues. We do provide a high level summary of all other current experiments in Appendix A. To accompany our experiment descriptions, we present and analysis comparing three baseline algorithms on bsuite: DQN (a), A2C and Bootstrapped DQN . As part of our open source effort, we include full code for these agents and more at bsuite/baselines. All plots and analysis are generated through the automated bsuite Jupyter notebook, and give a flavour for the sort of agent comparisons that are made easy by bsuite. Almost everyone agrees that a competent learning system requires memory, and almost everyone finds the concept of memory intuitive. Nevertheless, it can be difficult to provide a rigorous definition for memory. Even in human minds, there is evidence for distinct types of'memory' handled by distinct regions of the brain . The assessment of memory only becomes more difficult to analyse in the context of general learning algorithms, which may differ greatly from human models of cognition. Which types of memory should we analyse? How can we inspect belief models for arbitrary learning systems? Our approach in bsuite is to sidestep these debates through simple behavioural experiments. We refer to this experiment as memory length; it is designed to test the number of sequential steps an agent can remember a single bit. The underlying environment is based on a stylized T-maze , parameterized by a length N ∈ N. Each episode lasts N steps with observation o t = (c t, t/N) for t = 1,.., N and action space A = {−1, +1}. The context c 1 ∼ Unif(A) and c t = 0 for all t ≥ 2. The reward r t = 0 for all t < N, but r N = Sign(a N = c 1). For the bsuite experiment we run the agent on sizes N = 1,.., 100 exponentially spaced and look at the average regret compared to optimal after 10k episodes. The summary'score' is the percentage of runs for which the average regret is less than 75% of that achieved by a uniformly random policy. Memory length is a good bsuite experiment because it is targeted, simple, challenging, scalable and fast. By construction, an agent that performs well on this task has mastered some use of memory over multiple timesteps. Our summary'score' provides a quick and dirty way to compare agent performance at a high level. Our sweep over different lengths N provides empirical evidence about the scaling properties of the algorithm beyond a simple pass/fail. Figure 2a gives a quick snapshot of the performance of baseline algorithms. Unsurprisingly, actor-critic with a recurrent neural network greatly outperforms the feedforward DQN and Bootstrapped DQN. Figure 2b gives us a more detailed analysis of the same underlying data. Both DQN and Bootstrapped DQN are unable to learn anything for length > 1, they lack functioning memory. A2C performs well for all N ≤ 30 and essentially random for all N > 30, with quite a sharp cutoff. While it is not surprising that the recurrent agent outperforms feedforward architectures on a memory task, Figure 2b gives an excellent insight into the scaling properties of this architecture. In this case, we have a clear explanation for the observed performance: the RNN agent was trained via backprop-through-time with length 30. bsuite recovers an empirical evaluation of the scaling properties we would expect from theory. Reinforcement learning calls for a sophisticated form of exploration called deep exploration. Just as an agent seeking to'exploit' must consider the long term consequences of its actions towards cumulative rewards, an agent seeking to'explore' must consider how its actions can position it to learn more effectively in future timesteps. The literature on efficient exploration broadly states that only agents that perform deep exploration can expect polynomial sample complexity in learning . This literature has focused, for the most part, on bounding the scaling properties of particular algorithms in tabular MDPs through analysis . Our approach in bsuite is to complement this understanding through a series of behavioural experiments that highlight the need for efficient exploration. The deep sea problem is implemented as an N × N grid with a one-hot encoding for state. The agent begins each episode in the top left corner of the grid and descends one row per timestep. Each episode terminates after N steps, when the agent reaches the bottom row. In each state there is a random but fixed mapping between actions A = {0, 1} and the transitions'left' and'right'. At each timestep there is a small cost r = −0.01/N of moving right, and r = 0 for moving left. However, should the agent transition right at every timestep of the episode it will be rewarded with an additional reward of +1. This presents a particularly challenging exploration problem for two reasons. First, following the'gradient' of small intermediate rewards leads the agent away from the optimal policy. Second, a policy that explores with actions uniformly at random has probability 2 −N of reaching the rewarding state in any episode. For the bsuite experiment we run the agent on sizes N = 10, 12,.., 50 and look at the average regret compared to optimal after 10k episodes. The summary'score' computes the percentage of runs for which the average regret drops below 0.9 faster than the 2 N episodes expected by dithering. Deep Sea is a good bsuite experiment because it is targeted, simple, challenging, scalable and fast. By construction, an agent that performs well on this task has mastered some key properties of deep exploration. Our summary score provides a'quick and dirty' way to compare agent performance at a high level. Our sweep over different sizes N can help to provide empirical evidence of the scaling properties of an algorithm beyond a simple pass/fail. Figure 3 presents example output comparing A2C, DQN and Bootstrapped DQN on this task. Figure 4a gives a quick snapshot of performance. As expected, only Bootstrapped DQN, which was developed for efficient exploration, scores well. Figure 4b gives a more detailed analysis of the same underlying data. When we compare the scaling of learning with problem size N it is clear that only Bootstrapped DQN scales gracefully to large problem sizes. Although our experiment was only run to size 50, the regular progression of learning times suggest we might expect this algorithm to scale towards N > 50. This section describes some of the ways you can use bsuite in your research and development of RL algorithms. Our aim is to present a high-level description of some research and engineering use cases, rather than a tutorial for the code installation and use. We provide examples of specific investigations using bsuite in Appendixes C, D and E. Section 4 provides an outline of our code and implementation. Full details and tutorials are available at github.com/anon/bsuite. A bsuite experiment is defined by a set of environments and number of episodes of interaction. Since loading the environment via bsuite handles the logging automatically, any agent interacting with that environment will generate the data required for required for analysis through the Jupyter notebook we provide (Pérez &). Generating plots and analysis via the notebook only requires users to provide the path to the logged data. The'radar plot' (Figure 5) at the start of the notebook provides a snapshot of agent behaviour, based on summary scores. The notebook also contains a complete description of every experiment, summary scoring and in-depth analysis of each experiment. You can interact with the full report at bit.ly/bsuite-agents. If you are developing an algorithm to make progress on fundamental issues in RL, running on bsuite provides a simple way to replicate benchmark experiments in the field. Although many of these problems are'small', in the sense that their solution does not necessarily require large neural architecture, they are designed to highlight key challenges in RL. Further, although these experiments do offer a summary'score', the plots and analysis are designed to provide much more information than just a leaderboard ranking. By using this common code and analysis, it is easy to benchmark your agents and provide reproducible and verifiable research. If you are using RL as a tool to crack a'grand challenge' in AI, such as beating a world champion at Go, then taking on bsuite gridworlds might seem like small fry. We argue that one of the most valuable uses of bsuite is as a diagnostic'unit-test' for large-scale algorithm development. Imagine you believe that'better exploration' is key to improving your performance on some challenge, but when you try your'improved' agent, the performance does not improve. Does this mean your agent does not do good exploration? Or maybe that exploration is not the bottleneck in this problem? Worse still, these experiments might take days and thousands of dollars of compute to run, and even then the information you get might not be targeted to the key RL issues. Running on bsuite, you can test key capabilities of your agent and diagnose potential improvements much faster, and more cheaply. For example, you might see that your algorithm completely fails at credit assignment beyond n = 20 steps. If this is the case, maybe this lack of credit-assignment over long horizons is the bottleneck and not necessarily exploration. This can allow for much faster, and much better informed agent development -just like a good suite of tests for software development. Another benefit of bsuite is to disseminate your more easily and engage with the research community. For example, if you write a conference paper targeting some improvement to hierarchical reinforcement learning, you will likely provide some justification for your in terms of theorems or experiments targeted to this setting. 2 However, it is typically a large amount of work to evaluate your algorithm according to alternative metrics, such as exploration. This means that some fields may evolve without realising the connections and distinctions between related concepts. If you run on bsuite, you can automatically generate a one-page Appendix, with a link to a notebook report hosted online. This can help provide a scientific evaluation of your algorithmic changes, and help to share your in an easily-digestible format, compatible with ICML, ICLR and NeurIPS formatting. We provide examples of these experiment reports in Appendices B, C, D and E. To avoid discrepancies between this paper and the source code, we suggest that you take practical tutorials directly from github.com/anon/bsuite. A good starting point is bit.ly/bsuite-tutorial: a Jupyter notebook where you can play the code right from your browser, without installing anything. 3 The purpose of this section is to provide a high-level overview of the code that we open source. In particular, we want to stress is that bsuite is designed to be a library for RL research, not a framework. We provide implementations for all the environments, analysis, run loop and even baseline agents. However, it is not necessary that you make use of them all in order to make use of bsuite. The recommended method is to implement your RL agent as a class that implements a policy method for action selection, and an update method for learning from transitions and rewards. Then, simply pass your agent to our run loop, which enumerates all the necessary bsuite experiments and logs all the data automatically. If you do this, then all the experiments and analysis will be handled automatically and generate your via the included Jupyter notebook. We provide examples of running these scripts locally, and via Google cloud through our tutorials. If you have an existing codebase, you can still use bsuite without migrating to our run loop or agent structure. Simply replace your environment with environment = bsuite.load and record(bsuite id) and add the flag bsuite id to your code. You can then complete a full bsuite evaluation by iterating over the bsuite ids defined in sweep. SWEEP. Since the environments handle the logging themselves, your don't need any additional logging for the standard analysis. Although full bsuite includes many separate evaluations, no single bsuite environment takes more than 30 minutes to run and the sweep is naturally parallel. As such, we recommend launching in parallel using multiple processes or multiple machines. Our examples include a simple approach using Python's multiprocessing module with Google cloud compute. We also provide examples of running bsuite from OpenAI baselines and Dopamine . Designing a single RL agent compatible with diverse environments can cause problems, particularly for specialized neural networks. bsuite alleviates this problem by specifying an observation spec that surfaces the necessary information for adaptive network creation. By default, bsuite environments implement the dm env standards , but we also include a wrapper for use through Openai gym . However, if your agent is hardcoded for a format, bsuite offers the option to output each environment with the observation spec of your choosing via linear interpolation. This means that, if you are developing a network suitable for Atari and particular observation spec, you can choose to swap in bsuite without any changes to your agent. This paper introduces the Behaviour Suite for Reinforcement Learning, and marks the start of its ongoing development. With our opensource effort, we chose a specific collection of experiments as the bsuite2019 release, but expect this collection to evolve in future iterations. We are reaching out to researchers and practitioners to help collate the most informative, targeted, scalable and clear experiments possible for reinforcement learning. To do this, submissions should implement a sweep that determines the selection of environments to include and logs the necessary data, together with an analysis that parses this data. In order to review and collate these submissions we will be forming a bsuite committee. The committee will meet annually during the NeurIPS conference to decide which experiments will be included in the bsuite release. We are reaching out to a select group of researchers, and hope to build a strong core formed across industry and academia. If you would like to submit an experiment to bsuite or propose a committee member, you can do this via github pull request, or via email to bsuite.committee@gmail.com. This appendix outlines the experiments that make up the bsuite 2019 release. In the interests of brevity, we provide only an outline of each experiment here. Full documentation for the environments, interaction and analysis are kept with code at github.com/anon/bsuite. We begin with a collection of very simple decision problems, and standard analysis that confirms an agent's competence at learning a rewarding policy within them. We call these experiments'basic', since they are not particularly targeted at specific core issues in RL, but instead test a general base level of competence we expect all general agents to attain. To investigate the robustness of RL agents to noisy rewards, we repeat the experiments from Section A.1 under differing levels of Gaussian noise. This time we allocate the 20 different seeds across 5 levels of Gaussian noise N (0, σ 2) for σ = [0.1, 0.3, 1, 3, 10] with 4 seeds each. To investigate the robustness of RL agents to problem scale, we repeat the experiments from Section A.1 under differing reward scales. This time we allocate the 20 different seeds across 5 levels of reward scaling, where we multiply the observed rewards by λ = [0.01, 0.1, 1, 10, 100] with 4 seeds each. As an agent interacts with its environment, it observes the outcomes that from previous states and actions, and learns about the system dynamics. This leads to a fundamental tradeoff: by exploring poorly-understood states and actions the agent can learn to improve future performance, but it may attain better short-run performance by exploiting its existing knowledge. Exploration is the challenge of prioritizing useful information for learning, and the experiments in this section are designed to necessitate efficient exploration for good performance. A.4. If you run an agent on bsuite, and you want to share these as part of a conference submission, we make it easy to share a single-page'bsuite report' as part of your appendix. We provide a simple L A T E Xfile that you can copy/paste into your paper, and is compatible out-the-box with ICLR, ICML and NeurIPS style files. This single page summary displays the summary scores for experiment evaluations for one or more agents, with plots generated automatically from the included ipython notebook. In each report, two sections are left for the authors to fill in: one describing the variants of the agents examined and another to give some brief commentary on the . We suggest that authors promote more in-depth analysis to their main papers, or simply link to a hosted version of the full bsuite analysis online. You can find more details on our automated reports at github.com/anon/bsuite. The sections that follow are example bsuite reports, that give some example of how these report appendixes might be used. We believe that these simple reports can be a good complement to conference submissions in RL research, that'sanity check' the elementary properties of algorithmic implementations. An added bonus of bsuite is that it is easy to set up a like for like experiment between agents from different'frameworks' in a way that would be extremely laborious for an individual researcher. If you are writing a conference paper on a new RL algorithm, we believe that it makes sense for you to include a bsuite report in the appendix by default. The Behaviour Suite for Reinforcement Learning, or bsuite for short, is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent. The aim of the bsuite project is to collect clear, informative and scalable problems that capture key issues in the design of efficient and general learning algorithms and study agent behaviour through their performance on these shared benchmarks. This report provides a snapshot of agent performance on bsuite2019, obtained by running the experiments from github.com/anon/bsuite. In this experiment all implementations are taken from bsuite/baselines with default configurations. We provide a brief summary of the agents run on bsuite2019: • random: selects action uniformly at random each timestep. • dqn: Deep Q-networks Mnih et al. (2015b). • boot dqn: bootstrapped DQN with prior networks Osband et al. (2016; 2018). • actor critic rnn: an actor critic with recurrent neural network. Each bsuite experiment outputs a summary score in. We aggregate these scores by according to key experiment type, according to the standard analysis notebook. A detailed analysis of each of these experiments may be found in a notebook hosted on Colaboratory bit.ly/bsuite-agents. • random performs uniformly poorly, confirming the scores are working as intended. • dqn performs well on basic tasks, and quite well on credit assignment, generalization, noise and scale. DQN performs extremely poorly across memory and exploration tasks. The feedforward MLP has no mechanism for memory, and =5%-greedy action selection is inefficient exploration. • boot dqn is mostly identically to DQN, except for exploration where it greatly outperforms. This matches our understanding of Bootstrapped DQN as a variant of DQN designed to estimate uncertainty and use this to guide deep exploration. • actor critic rnn typically performs worse than either DQN or Bootstrapped DQN on all tasks apart from memory. This agent is the only one able to perform better than random due to its recurrent network architecture. The Behaviour Suite for Reinforcement Learning, or bsuite for short, is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent. The aim of the bsuite project is to collect clear, informative and scalable problems that capture key issues in the design of efficient and general learning algorithms and study agent behaviour through their performance on these shared benchmarks. This report provides a snapshot of agent performance on bsuite2019, obtained by running the experiments from github.com/anon/bsuite. All agents correspond to different instantiations of the DQN agent (b), as implemented in bsuite/baselines but with differnet optimizers from. In each case we tune a learning rate to optimize performance on'basic' tasks from {1e-1, 1e-2, 1e-3}, keeping all other parameters constant at default value. • sgd: vanilla stochastic gradient descent with learning rate 1e-2. • rmsprop: RMSProp with learning rate 1e-3. • adam: Adam with learning rate 1e-3. Each bsuite experiment outputs a summary score in. We aggregate these scores by according to key experiment type, according to the standard analysis notebook. A detailed analysis of each of these experiments may be found in a notebook hosted on Colaboratory: bit.ly/bsuite-optim. Both RMSProp and Adam perform better than SGD in every category. In most categories, Adam slightly outperforms RMSprop, although this difference is much more minor. SGD performs particularly badly on environments that require generalization and/or scale. This is not particularly surprising, since we expect the non-adaptive SGD may be more sensitive to learning rate optimization or annealing. In Figure 11 we can see that the differences are particularly pronounced on the cartpole domains. We hypothesize that this task requires more efficient neural network optimization, and the nonadaptive SGD is prone to numerical issues. The Behaviour Suite for Reinforcement Learning, or bsuite for short, is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent. The aim of the bsuite project is to collect clear, informative and scalable problems that capture key issues in the design of efficient and general learning algorithms and study agent behaviour through their performance on these shared benchmarks. This report provides a snapshot of agent performance on bsuite2019, obtained by running the experiments from github.com/anon/bsuite. In this experiment, all agents correspond to different instantiations of a Bootstrapped DQN with prior networks Osband et al. (2016; 2018). We take the default implementation from bsuite/baselines. We investigate the effect of the number of models used in the ensemble, sweeping over {1, 3, 10, 30}. Each bsuite experiment outputs a summary score in. We aggregate these scores by according to key experiment type, according to the standard analysis notebook. A detailed analysis of each of these experiments may be found in a notebook hosted on Colaboratory: bit.ly/bsuite-ensemble. Generally, increasing the size of the ensemble improves bsuite performance across the board. However, we do see signficantly decreasing returns to ensemble size, so that ensemble 30 does not perform much better than size 10. These are not predicted by the theoretical scaling of proven bounds , but are consistent with previous empirical findings;. The gains are most extreme in the exploration tasks, where ensemble sizes less than 10 are not able to solve large'deep sea' tasks, but larger ensembles solve them reliably. Even for large ensemble sizes, our implementation does not completely solve every cartpole swingup instance. Further examination learning curves suggests this may be due to some instability issues, which might be helped by using Double DQN to combat value overestimation van.

Represent the TLDR for finding the corresponding scientific article.

Bsuite is a collection of carefully-designed experiments that investigate the core capabilities of RL agents.

In this short paper, we briefly introduce the advantages of using AI planning in Cloud Migration, a preliminary prototype, as well as the chal- lenges the requires attention from the planning and schedul- ing society.


Question: Represent the scientific article to find a short TLDR

Question: Sequence-to-sequence attention-based models are a promising approach for end-to-end speech recognition. The increased model power makes the training procedure more difficult, and analyzing failure modes of these models becomes harder because of the end-to-end nature. In this work, we present various analyses to better understand training and model properties. We investigate on pretraining variants such as growing in depth and width, and their impact on the final performance, which leads to over 8% relative improvement in word error rate. For a better understanding of how the attention process works, we study the encoder output and the attention energies and weights. Our experiments were performed on Switchboard, LibriSpeech and Wall Street Journal. The encoder-decoder framework with attention BID34 BID60 has been successfully applied to automatic speech recognition (ASR) BID26 BID61 BID58 BID47 and is a promising end-to-end approach. The model outputs are words, sub-words or characters, and training the model can be done from scratch without any prerequisites except the training data in terms of audio features with corresponding transcriptions. In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach BID8 , ], the encoder-decoder model does not model the alignment explicitly. In the hybrid HMM/NN approach, a latent variable of hidden states is introduced, which model the phone state for any given time position. Thus by searching for the most probable sequence of hidden states, we get an explicit alignment. There is no such hidden latent variable in the encoder decoder model. Instead there is the attention process which can be interpreted as an implicit soft alignment. As this is only implicit and soft, it is harder to enforce constraints such as monotonicity, i.e. that the attention of future label outputs will focus also only to future time frames. Also, the interpretation of the attention weights as a soft alignment might not be completely valid, as the encoder itself can shift around and reorder evidence, i.e. the neural network could learn to pass over information in any possible way. E.g. the encoder could compress all the information of the input into a single frame and the decoder can learn to just attend on this single frame. We observed this behavior in early stages of the training. Thus, studying the temporal "alignment" behavior of the attention model becomes more difficult. Other end-to-end models such as connectionist temporal classification BID21 has often been applied to ASR in the past BID20 BID23 BID35 BID1 BID51 BID2 BID26 BID63 BID67. Other approaches are e.g. the inverted hidden Markov / segmental encoder-decoder model BID5, the recurrent transducer BID4 BID41, or the recurrent neural aligner. Depending on the interpretation, these can all be seen as variants of the encoder decoder approach. In some of these models, the attention process is not soft, but a hard decision. This hard decision can also become a latent variable such that we include several choices in the beam search. This is also referred to as hard attention. Examples of directly applying this idea on the usual attention approach are given by BID43, BID0,, BID33, BID27.We study recurrent NN (RNN) encoder decoder models in this work, which use long short-term memory (LSTM) units BID24. Recently the transformer model BID57 gained attention, which only uses feed-forward and self-attention layers, and the only recurrence is the label feedback in the decoder. As this does not include any temporal information, some positional encoding is added. This is not necessary for a RNN model, as it can learn such encoding by itself, which we demonstrate later for our attention encoder. We study attention models in more detail here. We are interested in when, why and how they fail and do an analysis on the search errors and relative error positions. We study the implicit alignment behavior via the attention weights and energies. We also analyze the encoder output representation and find that it contains information about the relative position and that it specially marks frames which should not be attended to, which correspond to silence.2 Related work BID25 analyzes individual neuron activations of a RNN language model and finds a neuron which becomes sensitive to the position in line. BID7 analyzed the hidden activations of the DeepSpeech 2 BID1 ] CTC end-to-end system and shows their correlation to a phoneme frame alignment. BID36 analyzed the encoder state and the attention weights of an attention model and makes similar observations as we do. Attention plots were used before to understand the behaviour of the model BID15. BID6 performed a comparison of the alignment behavior between hybrid HMM/NN models, the inverted HMM and attention models. BID42 investigate the effects of varying block sizes, attention types, and sub-word units. Understanding the inner working of a speech recognition system is also subject in, where the authors examine activation distribution and temporal patterns, focussing on the comparison between LSTM and GRU systems. A number of saliency methods BID50 BID32 BID52 are used for interpreting model decisions. In all cases, we use the RETURNN framework BID65 for neural network training and inference, which is based on TensorFlow BID54 and contains some custom CUDA kernels. In case of the attention models, we also use RETURNN for decoding. All experiments are performed on single GPUs, we did not take advantage of multi-GPU training. In some cases, the feature extraction, and in the hybrid case the decoding, is performed with RASR BID59. All used configs as well as used source code are published. The Switchboard corpus BID19 ] consists of English telephone speech. We use the 300h train dataset (LDC97S62), and a 90% subset for training, and a small part for cross validation, which is used for learning rate scheduling and to select a few models for decoding. We decode and report WER on Hub5'00 and Hub5'01. We use Hub5'00 to select the best model which we report the numbers on. Our hybrid HMM/NN model uses a deep bidirectional LSTM as described by BID64. Our baseline has 6 layers with 500 nodes in each direction. It uses dropout of 10% on the non-recurrent input of each LSTM layer, gradient noise with standard deviation of 0.3, Adam with Nesterov momentum (Nadam) BID18, Newbob learning rate scheduling BID64, and focal loss BID28 ].Our attention model uses byte pair encoding BID49 as subword units. We follow the baseline with about 1000 BPE units as described by. All our baselines and a comparison to from the literature are summarized in TAB0. The LibriSpeech dataset BID37 are read audio books and consists of about 1000h of speech. A subset of the training data is used for cross-validation, to perform learning rate scheduling and to select a number of models for full decoding. We use the dev-other set for selecting the final best model. The end-to-end attention model uses byte pair encoding (BPE) BID49 as subword units with a vocabulary of 10k BPE units. We follow the baseline as described by. A comparison of our baselines and other models are in TAB1. The Wall Street Journal (WSJ) dataset BID39 is read text from the WSJ. We use 90% of si284 for training, the remaining for cross validation and learning rate scheduling, dev93 for validation and selection of the final model, and eval92 for the final evaluation. We trained an end-to-end attention model using BPE subword units, with a vocabulary size of about 1000 BPE units. Our preliminary are shown in TAB2. Our attention model is based on the improved pretraining scheme as described in Section 5. the search errors where the models recognized sentence (via beam search) has a worse model score than the ground truth sentence. We observe that we do only very few search errors, and the amount of search errors seems independent from the final WER performance. Thus we conclude that we mostly have a problem in the model. We also were interested in the score difference between the best recognized sentence and the ground truth sentence. The are in Fig. 2. We can see that they concentrate on the lower side, around 10%, which is an indicator why a low beam size seems to be sufficient. It has been observed that pretraining can be substantial for good performance, and sometimes to get a converging model at all [a,b]. We provide a study on cases with attention-based models where pretraining benefits convergence, and compare the performance with and without pretraining. 8 >100 >100 >100 >100 6 8 >100 >100 >100 >100The pretraining variant of the Switchboard baseline (6 layers, time reduction 8 after pretraining) consists of these steps: 1. starts with 2 layers (layer 1 and 6), time reduction 32, and dropout as well as label smoothing disabled; 2. enable dropout; 3. 3 layers (layer 1, 2 and 6); 4. 4 layers (layer 1, 2, 3 and 6); 5. 5 layers (layer 1, 2, 3, 4 and 6); 6. all 6 layers; 7. decrease time reduction to 8; 8. final model, enable label smoothing. Each pretrain step is repeated for 5 epochs, where one epoch corresponds to 1/6 of the whole train corpus. In addition, a linear learning rate warm-up is performed from 1e-4 to 1e-3 in 10 epochs. We have to start with 2 layers as we want to have the time pooling in between the LSTM layers. In TAB3, performed on Switchboard, we varied the number of encoder layers and encoder LSTM units, both with and without pretraining. We observe that the overall best model is with 4 layers without the pretraining variant. I.e. we showed that we can directly start with 4 layers and time reduction 8 and yield very good . We even can start directly with 6 layer with a reduced learning rate. This was surprising to us, as this was not possible in earlier experiments. This might be due to a reduced and improved BPE vocabulary. We note that overall all the pretraining experiments seems to run more stable. We also can see that with 6 layers (and also more), pretraining yields better than no pretraining. These motivated us to perform further investigations into different variants of pretraining. It seems that pretraining allows to train deeper model, however using too much pretraining can also hurt. We showed that we can directly start with a deeper encoder and lower time reduction. In TAB4, we analyzed the optimal initial number of layers, and the initial time reduction. We observed that starting with a deeper network improves the overall performance, but also it still helps to then go deeper during pretraining, and starting too deep does not work well. We also observed that directly starting with time reduction 8 also works and further improves the final performance, but it seems that this makes the training slightly more unstable. In further experiments, we directly start with 4 layers and time reduction 8. We were also interested in the optimal number of repetitions of each pretrain step, i.e. how much epochs to train with each pretrain step; the baseline had 5 repetitions. We collected the in TAB5. In further experiments, we keep 5 repetitions as the default. It has already been shown by that a lower final time reduction performed better. So far, the lowest time reduction was 8 in our experiments. By having a pool size of 3 in the first time max pooling layer, we achieve a better-performing model with time reduction factor of 6 as shown in TAB6. So far we always kept the top layer (layer 6) during pretraining as our intuition was that it might help to get always the same time reduction factor as an input to this layer. When directly starting with the low time reduction, we do not need this scheme anymore, and we can always add a new layer on top. Comparisons are collected in TAB6. We can conclude that this simpler scheme to add layers on top performs better. We also did experiments with growing the encoder width / number of LSTM units during pretraining. We do this orthogonal to the growing in depth / number of layers. As before, our final number of LSTM units in each direction of the bidirectional deep LSTM encoder is 1024. Initially, we start with 50% of the final width, i.e. with 512 units. In each step, we linearly increase the number of units such that we have the final number of units in the last step. We keep the weights of existing units, and weights from/to newly added units are randomly initialized. We also decrease the dropout rate by the same factor. We can see that this width growing scheme performs better. This leads us to our current best model. Our findings are that pretraining is in general more stable, esp. for deep models. However, the pretraining scheme is important, and less pretraining can improve the performance, although it becomes more unstable. We also used the same improved pretraining scheme and time reduction 6 for WSJ as well as LibriSpeech and observed similar improvements, compare TAB1. We have observed that training attention models can be unstable, and careful tuning of initial learning rate, warm-up and pretraining is important. Related to that, we observe a high training variance. I.e. with the same configuration but different random seeds, we get some variance in the final WER performance. We observed this even for the same random seed, which we suspect stems from non-deterministic behaviour in TensorFlow operations such as tf.reduce_sum based on kernels 3-19.9, 19.6, 0.24 25.6-26.6, 26.1, 0.38 12.8-13.3, 13.1, 0.17 19.0-19.7, 19.4, 0.20 attention 5 runs 19.1-19.6, 19.3, 0.22 25.3-26.3, 25.8, 0.40 12.7-13.0, 12.9, 0.12 18.9-19.6, 19.2, 0.27 hybrid 5 seeds 14.3-14.5, 14.4, 0.08 19.0-19.3, 19.1, 0.12 9.6-9.8, 9.7, 0.06 14.3-14.7, 14.5, 0.16 hybrid 5 runs 14.3-14.5, 14.4, 0.07 19.0-19.2, 19.1, 0.09 9.6-9.8, 9.7, 0.08 14.4-14.6, 14.5 This training variance seems to be about the same as due to random seeds, which is higher than we expected. Note that it also depends a lot on other hyper parameters. For example, in a previous iteration of the model using a larger BPE vocabulary, we have observed more unstable training with higher variance, and even sometimes some models diverge while others converge with the same settings. We also compare that to hybrid HMM/LSTM models. It can be observed that it is lower compared to the attention model. We argue that is due to the more difficult optimization problem, and also due to the much bigger model. All the can be seen in TAB7. The encoder creates a high-level representation of the input. It also arguably represents further information needed for the decoder to know where to attend to. We try to analyze the output of the encoder and identify and examine the learned function. In FIG6, we plotted the encoder output and the attention weights, as well as the word positions in the audio. One hypothesis for an important function of the encoder is the detection of frames which should not be attended on by the decoder, e.g. which are silent or non-speech. Such a pattern can be observed in FIG6. By performing a dimensionality reduction (PCA) on the encoder output, we can identify the most important distinct information, which we identify as silence detection and encoder time position, compare FIG3. Similar behavior was shown by BID36. We further try to identify individual cells in the LSTM which encodes the positional information. By qualitatively inspecting the different neurons activations, we have identified multiple neurons which perform the hypothesized function as shown in FIG4 We also observed that the attention weights are always very local in the encoder frames, and often focus mostly on a single encoder frame, compare FIG6. The sharp behavior in the converged attention weight distribution has been observed before BID6 BID36. We conclude that the information about the label also needs to be well-localized in the encoder output. To support this observation, we performed experiments where we explicitly allowed only a local fixed-size window of non-zero attention weights around the arg max of the attention energies, to understand how much we can restrict the local context. The can be seen in TAB9. This confirms the hypothesis that the information is localized in the encoder. We explain the gap in performance with decoder frames where the model is unsure to attend, and where a global attention helps the decoder to gather information from multiple frames at once. We observed that in such case, there is sometimes some relatively large attention weight on the very first and/or very last frame. We provided an overview of our recent attention models on Switchboard, LibriSpeech and WSJ. We performed an analysis on the beam search errors. By our improved pretraining scheme, we improved our Switchboard baseline by over 8% relative in WER. We pointed out the high training variance of attention models compared to hybrid HMM/NN models. We analyzed the encoder output and identified the representation of the relative input position, both clearly visible in the PCA reduction of the encoder but even represented by individual neurons. Also we found indications that the encoder marks frames which can be skipped by decoder, which correlate to silence.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

improved pretraining, and analysing encoder output and attention

We compare many tasks and task combinations for pretraining sentence-level BiLSTMs for NLP tasks. Language modeling is the best single pretraining task, but simple baselines also do well.