Question: Represent the article for finding a single-sentence TLDR.

Question: Combining deep model-free reinforcement learning with on-line planning is a promising approach to building on the successes of deep RL. On-line planning with look-ahead trees has proven successful in environments where transition models are known a priori. However, in complex environments where transition models need to be learned from data, the deficiencies of learned models have limited their utility for planning. To address these challenges, we propose TreeQN, a differentiable, recursive, tree-structured model that serves as a drop-in replacement for any value function network in deep RL with discrete actions. TreeQN dynamically constructs a tree by recursively applying a transition model in a learned abstract state space and then aggregating predicted rewards and state-values using a tree backup to estimate Q-values. We also propose ATreeC, an actor-critic variant that augments TreeQN with a softmax layer to form a stochastic policy network. Both approaches are trained end-to-end, such that the learned model is optimised for its actual use in the tree. We show that TreeQN and ATreeC outperform n-step DQN and A2C on a box-pushing task, as well as n-step DQN and value prediction networks on multiple Atari games. Furthermore, we present ablation studies that demonstrate the effect of different auxiliary losses on learning transition models. A promising approach to improving model-free deep reinforcement learning (RL) is to combine it with on-line planning. The model-free value function can be viewed as a rough global estimate which is then locally refined on the fly for the current state by the on-line planner. Crucially, this does not require new samples from the environment but only additional computation, which is often available. One strategy for on-line planning is to use look-ahead tree search BID12 BID2. Traditionally, such methods have been limited to domains where perfect environment simulators are available, such as board or card games BID4 BID24. However, in general, models for complex environments with high dimensional observation spaces and complex dynamics must be learned from agent experience. Unfortunately, to date, it has proven difficult to learn models for such domains with sufficient fidelity to realise the benefits of look-ahead planning BID17 BID29.A simple approach to learning environment models is to maximise a similarity metric between model predictions and ground truth in the observation space. This approach has been applied with some success in cases where model fidelity is less important, e.g., for improving exploration BID3 BID17. However, this objective causes significant model capacity to be devoted to predicting irrelevant aspects of the environment dynamics, such as noisy s, at the expense of value-critical features that may occupy only a small part of the observation space (Pathak et al., Since the transition model is only weakly grounded in the actual environment, our approach can alternatively be viewed as a model-free method in which the fully connected layers of DQN are replaced by a recursive network that applies transition functions with shared parameters at each tree node expansion. The ing architecture, which we call TreeQN, encodes an inductive bias based on the prior knowledge that the environment is a stationary Markov process, which facilitates faster learning of better policies. We also present an actor-critic variant, ATreeC, in which the tree is augmented with a softmax layer and used as a policy network. We show that TreeQN and ATreeC outperform their DQN-based counterparts in a box-pushing domain and a suite of Atari games, with deeper trees often outperforming shallower trees, and TreeQN outperforming VPN BID18 on most Atari games. We also present ablation studies investigating various auxiliary losses for grounding the transition model more strongly in the environment, which could improve performance as well as lead to interpretable internal plans. While we show that grounding the reward function is valuable, we conclude that how to learn strongly grounded transition models and generate reliably interpretable plans without compromising performance remains an open research question. We consider an agent learning to act in a Markov Decision Process (MDP), with the goal of maximising its expected discounted sum of rewards R t = ∞ t=0 γ t r t, by learning a policy π(s) that maps states s ∈ S to actions a ∈ A. The state-action value function (Q-function) is defined as DISPLAYFORM0 The Bellman optimality equation writes Q * recursively as DISPLAYFORM1 where P is the MDP state transition function and r is a reward function, which for simplicity we assume to be deterministic. Q-learning BID33 ) uses a single-sample approximation of the contraction operator T to iteratively improve an estimate of Q *.In deep Q-learning BID14, Q is represented by a deep neural network with parameters θ, and is improved by regressing Q(s, a) to a target r + γ max a Q(s, a ; θ −), where θ − are the parameters of a target network periodically copied from θ. We use a version of n-step Q-learning BID15 with synchronous environment threads. In particular, starting at a timestep t, we roll forward n env = 16 threads for n = 5 timesteps each. We then bootstrap off the final states only and gather all n env × n = 80 transitions in a single batch for the backward pass, minimising the loss: DISPLAYFORM2 If the episode terminates, we use the remaining episode return as the target, without bootstrapping. This algorithm's actor-critic counterpart is A2C, a synchronous variant of A3C BID15 in which a policy π and state-value function V (s) are trained using the gradient: DISPLAYFORM3 where A j is an advantage estimate given by DISPLAYFORM4, H is the policy entropy, β is a hyperparameter tuning the degree of entropy regularisation, and α is a hyperparameter controlling the relative learning rates of actor and critic. These algorithms were chosen for their simplicity and reasonable wallclock speeds, but TreeQN can also be used in other algorithms, as described in Section 3. Our implementations are based on OpenAI Baselines BID11 ). The canonical neural network architecture in deep RL with visual observations has a series of convolutional layers followed by two fully connected layers, where the final layer produces one output for each action-value. We can think of this network as first calculating an encoding z t of the state s t which is then evaluated by the final layer to estimate Q * (s t, a) (see FIG0).In tree-search on-line planning, a look-ahead tree of possible future states is constructed by recursively applying an environment model. These states are typically evaluated by a heuristic, a learned value function, or Monte-Carlo rollouts. Backups through the tree aggregate these values along with the immediate rewards accumulated along each path to estimate the value of taking an action in the current state. This paper focuses on a simple tree-search with a deterministic transition function and no value uncertainty estimates, but our approach can be extended to tree-search variants like UCT BID13 if the components remain differentiable. In this section, we propose TreeQN, a novel end-to-end differentiable tree-structured architecture for deep reinforcement learning. We first give an overview of the architecture, followed by details of each model component and the training procedure. TreeQN uses a recursive tree-structured neural network between the encoded state z t and the predicted state-action values Q(s t, a), instead of directly estimating the state-action value from the current encoded state z t using fully connected layers as in DQN BID14. Specifically, TreeQN uses a recursive model to refine its estimate of Q(s t, a) via learned transition, reward, and value functions, and a tree backup (see FIG1). Because these learned components are shared throughout the tree, TreeQN implements an inductive bias, missing from DQN, that reflects the prior knowledge that the Q-values are properties of a stationary Markov process. We also encode the inductive bias that Q-values may be expressed as a sum of scalar rewards and values. Specifically, TreeQN learns an action-dependent transition function that, given a state representation z l|t, predicts the next state representation z ai l+1|t for action a i ∈ A, and the corresponding rewardr ai l|t. To make the distinction between internal planning steps and steps taken in the environment explicit, we write z l|t to denote the encoded state at time t after l internal transitions, starting with z 0|t for the encoding of s t. TreeQN applies this transition function recursively to construct a tree containing the state representations and rewards received for all possible sequences of actions up to some predefined depth d ("Tree Transitioning" in FIG1 The value of each predicted state V (z) is estimated with a value function module. Using these values and the predicted rewards, TreeQN then performs a tree backup, mixing the k-step returns along each path in the tree using TD(λ) BID25 BID27. This corresponds to "Value Prediction & Backup" in FIG1 and can be formalized as DISPLAYFORM0 DISPLAYFORM1 where b is a function to recursively perform the backup. For 0 < λ < 1, value estimates of the intermediate states are mixed into the final Q-estimate, which encourages the intermediate nodes of the tree to correspond to meaningful states, and reduces the impact of outlier values. When λ = 1, and b is the standard hard max function, then Eq. 3 simplifies to a backup through the tree using the familiar Bellman equation: DISPLAYFORM2 We note that even for a tree depth of only one, TreeQN imposes a significant structure on the value function by decomposing it as a sum of action-conditional reward and next-state value, and using a shared value function to evaluate each next-state representation. Crucially, during training we backpropagate all the way from the final Q-estimate, through the value prediction, tree transitioning, and encoding layers of the tree, i.e., the entire network shown in FIG1. Learning these components jointly ensures that they are useful for planning on-line. In this section, we describe each of TreeQN's components in more detail. Encoder function. As in DQN, a series of convolutional layers produces an embedding of the observed state, z 0|t = encode(s t).Transition function. We first apply a single fully connected layer to the current state embedding, shared by all actions. This generates an intermediate representation (z env l+1|t) that could carry information about action-agnostic changes to the environment. In addition, we use a fully connected layer per action, which is applied to the intermediate representation to calculate a next-state representation that carries information about the effect of taking action a i. We use residual connections for these layers: DISPLAYFORM0 where DISPLAYFORM1 are learnable parameters. Note that the next-state representation is calculated for every action a i independently using the respective transition matrix W ai, but this transition function is shared for the same action throughout the tree. A caveat is that the model can still learn to use different parts of the latent state space in different parts of the tree, which could undermine the intended parameter sharing in the model structure. To help TreeQN learn useful transition functions that maintain quality and diversity in their latent states, we introduce a unit-length projection of the state representations by simply dividing a state's vector representation by its L2 norm before each application of the transition function, z l|t:= z l|t / z l|t. This prevents the magnitude of the representation from growing or shrinking, which encourages the behaviour of the transition function to be more consistent throughout the tree. Reward function. In addition to predicting the next state, we also predict the immediate reward for every action a i ∈ A in state z l|t usinĝ DISPLAYFORM2 where DISPLAYFORM3 and ReLU is the rectified linear unit BID16, and the predicted reward for a particular actionr ai l|t is the i-th element of the vectorr(z l|t). Value function. The value of a state representation z is estimated as DISPLAYFORM4 where DISPLAYFORM5 Backup function. We use the following function that can be recursively applied to calculate the tree backup: DISPLAYFORM6 Using a hard max for calculating the backup would in gradient information only being used to update parameters along the maximal path in the tree. By contrast, the softmax allows us to use downstream gradient information to update parameters along all paths. Furthermore, it potentially reduces the impact of outlier value predictions. With a learned temperature for the softmax, this function could represent the hard max arbitrarily closely. However, we did not find an empirical difference so we left the temperature at 1. The TreeQN architecture is fully differentiable, so we can directly use it in the place of a Q-function in any deep RL algorithm with discrete actions. Differentiating through the entire tree ensures that the learned components are useful for planning on-line, as long as that planning is performed in the same way as during training. However, it seems plausible that auxiliary objectives based on minimising the error in predicting rewards or observations could improve the performance by helping to ground the transition and reward functions to the environment. It could also encourage TreeQN to perform model-based planning in an interpretable manner. In principle, such objectives could give rise to a spectrum of methods from model-free to fully model-based. At one extreme, TreeQN without auxiliary objectives can be seen as a model-free approach that draws inspiration from tree-search planning to encode valuable inductive biases into the neural network architecture. At the other extreme, perfect, grounded reward and transition models could in principle be learned. Using them in our architecture would then correspond to standard model-based lookahead planning. The sweet spot could be an intermediate level of grounding that maintains the flexibility of end-to-end model-free learning while benefiting from the additional supervision of explicit model learning. To investigate this spectrum, we experiment with two auxiliary objectives. Reward grounding. We experiment with an L2 loss regressingr a t:t+l−1 l|t, the predicted reward at level l of the tree corresponding to the selected action sequence {a t . . . a t+l−1}, to the true observed rewards. For each of the n timesteps of n-step Q-learning this gives: DISPLAYFORM0 where η r is a hyperparameter weighting the loss, andd = min(d, n − j + 1) restricts the sum to rewards for which we have already observed the true value. State grounding. We experiment with a grounding in the latent space, using an L2 loss to regress the predicted latent state z a t:t+l l|t at level l of the tree to z 0|t+l, the initial encoding of the true state corresponding to the actions actually taken: DISPLAYFORM1 By employing an additional decoder module, we could use a similar loss to regress decoded observations to the true observations. In informal experiments, joint training with such a decoder loss did not yield good performance, as also found by BID18.In Section 7.1, we present on the use these objectives, showing that reward grounding gives better performance, but that our method for state grounding does not. The intuitions guiding the design of TreeQN are as applicable to policy search as to valuebased RL, in that a policy can use a tree planner to improve its estimates of the optimal action probabilities BID8 BID22. As our proposed architecture is trained end-to-end, it can be easily adapted for use as a policy network. In particular, we propose ATreeC, an actor-critic extension of TreeQN. In this architecture, the policy network is identical to TreeQN, with an additional softmax layer that converts the Q estimates into the probabilities of a stochastic policy. The critic shares the encoder parameters, and predicts a scalar state value with a single fully connected layer: V cr (s) = w cr z + b cr. We used different parameters for the critic value function and the actor's tree-value-function module, but found that sharing these parameters had little effect on performance. The entire setup, shown in FIG2, is trained with A2C as described in Section 2, with the addition of the same auxiliary losses used for TreeQN. Note that TreeQN could also be used in the critic, but we leave this possibility to future work. There is a long history of work combining model-based and model-free RL. An early example is Dyna-Q BID26 ) which trains a model-free algorithm with samples drawn from a learned model. Similarly, van train a sparse model with some environment samples that can be used to refine a model-free Q-function. BID9 use local linear models to generate additional samples for their model-free algorithm. However, these approaches do not attempt to use the model on-line to improve value estimates. In deep RL, value iteration networks BID30 use a learned differentiable model to plan on the fly, but require planning over the full state space, which must also possess a spatial structure with local dynamics such that convolution operations can execute the planning algorithm. The predictron BID23 instead learns abstract-state transition functions in order to predict values. However, it is restricted to policy evaluation without control. Value prediction networks take a similar approach but are more closely related to our work because the learned model components are used in a tree for planning. However, in their work this tree is only used to construct targets and choose actions, and not to compute the value estimates during training. Such estimates are instead produced from non-branching trajectories following on-policy action sequences. By contrast, TreeQN is a unified architecture that constructs the tree dynamically at every timestep and differentiates through it, eliminating any mismatch between the model at training and test time. Furthermore, we do not use convolutional transition functions, and hence do not impose spatial structure on the latent state representations. These differences simplify training, allow our model to be used more flexibly in other training regimes, and explain in part our substantially improved performance on the Atari benchmark. BID6 propose differentiating through a stochastic programming optimisation using a probabilistic model to learn model parameters with respect to their true objective rather than a maximum likelihood surrogate. However, they do not tackle the full RL setting, and do not use the model to repeatedly or recursively refine predictions. Imagination-augmented agents BID34 learn to improve policies by aggregating rollouts predicted by a model. However, they rely on pretraining an observation-space model, which we argue will scale poorly to more complex environments. Further, their aggregation of rollout trajectories takes the form of a generic RNN rather than a value function and tree backup, so the inductive bias based on the structure of the MDP is not explicitly present. A class of value gradient methods BID5 BID7 BID10 also differentiates through models to train a policy. However, this approach does not use the model during execution to refine the policy, and requires continuous action spaces. BID17 and BID3 propose methods for learning observation-prediction models in the Atari domain, but use these models only to improve exploration. Variants of scheduled sampling BID1 may be used to improve robustness of these models, but scaling to complex domains has proven challenging BID28. We evaluate TreeQN and ATreeC in a simple box-pushing environment, as well as on the subset of nine Atari environments that BID18 use to evaluate VPN. The experiments are designed to determine whether or not TreeQN and ATreeC outperform DQN, A2C, and VPN, and whether they can scale to complex domains. We also investigate how to best ground the the transition function with auxiliary losses. Furthermore, we compare against alternative ways to increase the number of parameters and computations of a standard DQN architecture, and study the impact of tree depth. Full details of the experimental setup, as well as architecture and training hyperparameters, are given in the appendix. Grounding. We perform a hyperparameter search over the coefficients η r and η s of the reward and state grounding auxiliary losses, on the Atari environment Seaquest. These experiments aim to determine the relevant trade-offs between the flexibility of a model-free approach and the potential benefits of a more model-based algorithm. Box Pushing. We randomly place an agent, 12 boxes, 5 goals and 6 obstacles on the center 6 × 6 tiles of an 8 × 8 grid. The agent's goal is to push boxes into goals in as few steps as possible while avoiding obstacles. Boxes may not be pushed into each other. The obstacles, however, are'soft' in that they are do not block movement, but generate a negative reward if the agent or a box moves onto an obstacle. This rewards better planning without causing excessive gridlock. This environment is inspired by Sokoban, as used by BID34, in that poor actions can generate irreversibly TreeQN adds additional parameters to a standard DQN architecture. We compare TreeQN to two baseline architectures with increased computation and numbers of parameters to verify the benefit of the additional structure and grounding. DQN-Wide doubles the size of the embedding dimension (1024 instead of 512). DQN-Deep inserts two additional fully connected layers with shared parameters and residual connections between the two fully-connected layers of DQN. This is in effect a non-branching version of the TreeQN architecture that also lacks explicit reward prediction. In this section, we present our experimental for TreeQN and ATreeC.7.1 GROUNDING FIG4 shows the of a hyperparameter search on η r and η s, the coefficients of the auxiliary losses on the predicted rewards and latent states. An intermediate value of η r helps performance but there is no benefit to using the latent space loss. Subsequent experiments use η r = 1 and η s = 0.The predicted rewards that the reward-grounding objective encourages the model to learn appear both in its own Q-value prediction and in the target for n-step Q-learning. Consequently, we expect this auxiliary loss to be well aligned with the true objective. By contrast, the state-grounding loss (and other potential auxiliary losses) might help representation learning but would not explicitly learn any part of the desired target. It is possible that this mismatch between the auxiliary and primary objective leads to degraded performance when using this form of state grounding. One potential route to overcoming this obstacle to joint training would be pre-training a model, as done by BID34. Inside TreeQN this model could then be fine-tuned to perform well inside the planner. We leave this possiblity to future work. FIG3 shows the of TreeQN with tree depths 1, 2, and 3, compared to a DQN baseline. In this domain, there is a clear advantage for the TreeQN architecture over DQN. TreeQN learns policies that are substantially better at avoiding obstacles and lining boxes up with goals so they can be easily pushed in later. TreeQN also substantially speeds up learning. We believe that the greater structure brought by our architecture regularises the model, encouraging appropriate state representations to be learned quickly. Even a depth-1 tree improves performance significantly, as disentangling the estimation of rewards and next-state values makes them easier to learn. This is further facilitated by the sharing of value-function parameters across branches. When trained with n-step Q-learning, the deeper depth-2 and depth-3 trees learn faster and plateau higher than the shallow depth-1 tree. In the this domain, useful transition functions are relatively easy to learn, and the extra computation time with those transition modules can help refine value estimates, yielding advantages for additional depth. FIG6 shows the of ATreeC with tree depths 1, 2, and 3, compared to an A2C baseline. As with TreeQN, ATreeC substantially outperforms the baseline. Furthermore, thanks to its stochastic policy, it substantially outperforms TreeQN. Whereas TreeQN and DQN sometimes indecisively bounce back and forth between adjacent states, ATreeC captures this uncertainty in its policy probabilities and thus acts more decisively. However, unlike TreeQN, ATreeC shows no pronounced differences for different tree depths. This is in part due to a ceiling effect in this domain. However, ATreeC is also gated by the quality of the critic's value function, which in these experiments was a single linear layer after the state encoding as described in Section 4. Nonetheless, this demonstrates the ease with which TreeQN can be used as a drop-in replacement for any deep RL algorithm that learns policies or value functions for discrete actions.7.3 ATARI TAB1 summarises all our Atari , while FIG8 shows learning curves in depth. TreeQN shows substantial benefits in many environments compared to our DQN baseline, which itself often outperforms VPN BID18. ATreeC always matches or outperforms A2C. We present the mean performance of five random seeds, while the VPN reported by BID18, shown as dashed lines in FIG8, are the mean of the best five seeds of an unspecified number of trials. TreeQN. In all environments except Frostbite, TreeQN outperforms DQN on average, with the most significant gains in Alien, CrazyClimber, Enduro, Krull, and Seaquest. Many of these environments seem well suited to short horizon look-ahead planning, with simple dynamics that generalise well and tradeoffs between actions that become apparent only after several timesteps. For example, an incorrect action in Alien can trap the agent down a corridor with an alien. In Seaquest, looking ahead could help determine whether it is better to go deeper to collect more points or to surface for oxygen. However, even in a game with mostly reactive decisions like the racing game Enduro, TreeQN shows significant benefits. TreeQN also outperforms the additional baselines of DQN-Wide and DQN-Deep, indicating that the additional structure and grounding of our architecture brings benefits beyond simply adding model capacity and computation. In particular, it is interesting that DQN-Deep is often outperformed by the vanilla DQN baseline, as optimisation difficulties grow with depth. In contrast, the additional BID18 report the same statistic, but average instead over the best five of an unspecified number of agents.structure and auxiliary loss employed by TreeQN turn its additional depth from a liability into a strength. ATreeC. ATreeC matches or outperforms its baseline (A2C) in all environments. Compared to TreeQN, ATreeC's performance is better across most environments, particularly on Qbert, reflecting an overall advantage for actor-critic also found by BID15 and in our box-pushing experiments. However, performance is much worse on Seaquest, revealing a deficiency in exploration as policy entropy collapses too rapidly and consequently the propensity of policy gradient methods to become trapped in a local optimum. In Krull and Frostbite, most algorithms have poor performance, or high variance in returns from run to run, as agents are gated by their ability to explore. Both of these games require the completion of sub-levels in order to accumulate large scores, and none of our agents reliably explore beyond the initial stages of the game. Mean performance appears to favor TreeQN and ATreeC in Krull, and perhaps DQN in Frostbite, but the returns are too variable to draw from this number of random seeds. Combining TreeQN and ATreeC with smart exploration mechanisms is an interesting direction for future work to improve robustness of training in these types of environments. Compared to the box-pushing domain, there is less of a clear performance difference between trees of different depths. In some environments (Amidar, MsPacman), greater depth does appear to be employed usefully by TreeQN to a small extent, ing in the best-performing individual agents. However, for the Atari domain the embedding size for the transition function we use is much larger (512 compared to 128), and the dynamics are much more complex. Consequently, we expect that optimisation difficulties, and the challenge of learning abstract-state transition functions, impede the utility of deeper trees in some cases. We look to future work to further refine methods for learning to plan abstractly in complex domains. However, the decomposition of Q-value into reward and next-state value employed by the first tree expansion is clearly of utility in a broad range of tasks. When inspecting the learned policies and trees, we find that the values sometimes correspond to intuitive reasoning about sensible policies, scoring superior action sequences above poorer ones. However, we find that the actions corresponding to branches of the tree that are scored most highly are frequently not taken in future timesteps. The flexibility of TreeQN and ATreeC allows our agents to find any useful way to exploit the computation in the tree to refine action-value estimates. As we found no effective way to strongly ground the model components without sacrificing performance, the interpretability of learned trees is limited. We presented TreeQN and ATreeC, new architectures for deep reinforcement learning in discreteaction domains that integrate differentiable on-line tree planning into the action-value function or policy. Experiments on a box-pushing domain and a set of Atari games show the benefit of these architectures over their counterparts, as well as over VPN. In future work, we intend to investigate enabling more efficient optimisation of deeper trees, encouraging the transition functions to produce interpretable plans, and integrating smart exploration. A.1 BOX PUSHING Environment. For each episode, a new level is generated by placing an agent, 12 boxes, 5 goals and 6 obstacles in the center 6 × 6 tiles of an 8 × 8 grid, sampling locations uniformly. The outer tiles are left empty to prevent initial situations where boxes cannot be recovered. The agent may move in the four cardinal directions. If the agent steps off the grid, the episode ends and the agent receives a penalty of −1. If the agent moves into a box, it is pushed in the direction of movement. Moving a box out of the grid generates a penalty of −0.1. Moving a box into another box is not allowed and trying to do so generates a penalty of −0.1 while leaving all positions unchanged. When a box is pushed into a goal, it is removed and the agent receives a reward of +1.Obstacles generate a penalty of −0.2 when the agent or a box is moved onto them. Moving the agent over goals incurs no penalty. Lastly, at each timestep the agent receives a penalty of −0.01. Episodes terminate when 75 timesteps have elapsed, the agent has left the grid, or no boxes remain. The observation is given to the model as a tensor of size 5 × 8 × 8. The first four channels are binary encodings of the position of the agent, goals, boxes, and obstacles respectively. The final channel is filled with the number of timesteps remaining (normalised by the total number of timesteps allowed).Architecture. The encoder consists of (conv-3x3-1-24, conv-3x3-1-24, conv-4x4-1-48, fc-128), where conv-wxh-s-n denotes a convolution with n filters of size w × h and stride s, and fc-h denotes a fully connected layer with h hidden units. All layers are separated with ReLU nonlinearities. The hidden layer of the reward function MLP has 64 hidden units. Preprocessing of inputs follows the procedure of BID14, including concatenation of the last four frames as input, although we use a frameskip of 10.Architecture. The Atari experiments have the same architecture as for box-pushing, except for the encoder architecture which is as follows: (conv-8x8-4-16, conv-4x4-2-32, fc-512). All experiments use RMSProp BID31 ) with a learning rate of 1e-4, a decay of α = 0.99, and = 1e-5.The learning rate was tuned coarsely by running DQN on the Seaquest environment, and kept the same for all subsequent experiments (box-pushing and Atari).For DQN and TreeQN, for -greedy exploration was decayed linearly from 1 to 0.05 over the first 4 million environment transitions observed (after frameskipping, so over 40 million atomic Atari timesteps).For A2C and ATreeC, we use a value-function loss coefficient α = 0.5 and an entropy regularisation β = 0.01.The reward prediction loss was scaled by η r = 1.We use n steps = 5 and n envs = 16, for a total batch size of 80.The discount factor is γ = 0.99 and the target networks are updated every 40, 000 environment transitions.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present TreeQN and ATreeC, new architectures for deep reinforcement learning in discrete-action domains that integrate differentiable on-line tree planning into the action-value function or policy.

Represent the TLDR for finding the corresponding scientific article

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.


Question: Represent the article for finding a single-sentence TLDR

Question: Multi-label classification (MLC) is the task of assigning a set of target labels for a given sample. Modeling the combinatorial label interactions in MLC has been a long-haul challenge. Recurrent neural network (RNN) based encoder-decoder models have shown state-of-the-art performance for solving MLC. However, the sequential nature of modeling label dependencies through an RNN limits its ability in parallel computation, predicting dense labels, and providing interpretable . In this paper, we propose Message Passing Encoder-Decoder (MPED) Networks, aiming to provide fast, accurate, and interpretable MLC. MPED networks model the joint prediction of labels by replacing all RNNs in the encoder-decoder architecture with message passing mechanisms and dispense with autoregressive inference entirely. The proposed models are simple, fast, accurate, interpretable, and structure-agnostic (can be used on known or unknown structured data). Experiments on seven real-world MLC datasets show the proposed models outperform autoregressive RNN models across five different metrics with a significant speedup during training and testing time. Multi-label classification (MLC) is receiving increasing attention in tasks such as text categorization and image classification. Accurate and scalable MLC methods are in urgent need for applications like assigning topics to web articles, classifying objects in an image, or identifying binding proteins on DNA. The most common and straightforward MLC method is the binary relevance (BR) approach that considers multiple target labels independently BID0. However, in many MLC tasks there is a clear dependency structure among labels, which BR methods ignore. Accordingly, probabilistic classifier chain (PCC) models were proposed to model label dependencies and formulate MLC in an autoregressive sequential prediction manner BID1. One notable work in the PCC category was from which implemented a classifier chain using a recurrent neural network (RNN) based sequence to sequence (Seq2Seq) architecture, Seq2Seq MLC. This model uses an encoder RNN encoding elements of an input sequence, a decoder RNN predicting output labels one after another, and beam search that computes the probability of the next T predictions of labels and then chooses the proposal with the max combined probability. However, the main drawback of classifier chain models is that their inherently sequential nature precludes parallelization during training and inference. This can be detrimental when there are a large number of positive labels as the classifier chain has to sequentially predict each label, and often requires beam search to obtain the optimal set. Aside from time-cost disadvantages, PCC methods have several other drawbacks. First, PCC methods require a defined ordering of labels for the sequential prediction, but MLC output labels are an unordered set, and the chosen order can lead to prediction instability. Secondly, even if the optimal ordering is known, PCC methods struggle to accurately capture long-range dependencies among labels in cases where the number of positive labels is large (i.e., dense labels). For example, the Delicious dataset has a median of 19 positive labels per sample, so it can be difficult to correctly predict the labels at the end of the prediction chain. Lastly, many real-world applications prefer interpretable predictors. For instance, in the task of predicting which proteins (labels) will bind to a DNA sequence based binding site, users care about how a prediction is made and how the interactions among labels influence the predictions 1.Message Passing Neural Networks (MPNNs) BID3 introduce a class of methods that model joint dependencies of variables using neural message passing rather than an explicit representation such as a probabilistic classifier chain. Message passing allows for efficient inference by modelling conditional independence where the same local update procedure is applied iteratively to propagate information across variables. MPNNs provide a flexible method for modeling multiple variables jointly which have no explicit ordering (and can be modified to incorporate an order, as explained in section 3). To handle the drawbacks of BR and PCC methods, we propose a modified version of MPNNs for MLC by modeling interactions between labels using neural message passing. We introduce Message Passing Encoder-Decoder (MPED) Networks aiming to provide fast, accurate, and interpretable multi-label predictions. The key idea is to replace RNNs and to rely on neural message passing entirely to draw global dependencies between input components, between labels and input components, and between labels. The proposed MPED networks allow for significantly more parallelization in training and testing. The main contributions of this paper are:• Novel approach for MLC. To the authors' best knowledge, MPED is the first work using neural message passing for MLC.• Accurate MLC. Our model achieves similar, or better performance compared to the previous state of the art across five different MLC metrics. We validate our model on seven MLC datasets which cover a wide spectrum of input data structure: sequences (English text, DNA), tabular (bag-of-words), and graph (drug molecules), as well as output label structure: unknown and graph.• Fast. Empirically our model achieves an average 1.7x speedup over the autoregressive seq2seq MLC at training time and an average 5x speedup over its testing time.• Interpretable. Although deep-learning based systems have widely been viewed as "black boxes" due to their complexity, our attention based MPED models provide a straightforward way to explain label to label, input to label, and feature to feature dependencies. Message Passing Neural Networks (MPNNs) BID3 are a generalization of graph neural networks (GNNs) BID4 BID5, where variables are represented as nodes on a graph G and joint dependencies are modelled using message passing rather than explicit representations, which allows for efficient inference. MPNNs model the joint dependencies using message function M t and node update function U t for T time steps, where t is the current time step. The hidden state v t i ∈ R d of node i ∈ G is updated based on messages m t i from its neighboring nodes {v t j∈N (i) } defined by neighborhood N (i): DISPLAYFORM0 DISPLAYFORM1 After T updates, a readout function R is used on the updated nodes for a prediction (e.g., node classification or graph classification) on the graph G.Many possibilities exist for functions M t and U t. For example, one can pass messages using neural attention in which nodes are able to attend over their neighborhoods differentially BID6. This allows for the network to learn different weights for different nodes in a neighborhood, without depending on knowing the graph structure a priori. In this formulation, messages for node v t i are obtained by a weighted sum of all neighboring nodes {v t j∈N (i) } where the weights are obtained by attention BID7. In our implementation, we implement neural message passing with attention. In the rest of the paper, we use "graph attention" and "neural message passing" interchangeably. Neural message passing with attention works as follows. DISPLAYFORM2 where e t ij represents the importance of node j for node i. DISPLAYFORM3 Attention coefficients e t ij are then normalized across all neighboring nodes of node i using a softmax function: DISPLAYFORM4.In our method, we use a so called attention message function M t atn to produce the message from node j to node i using the learned attention weights α t ij and another transformation matrix W v ∈ R d×d. Then we compute the full message m t i by linearly combining messages from all nodes j ∈ N (i) with a residual connection on the current v DISPLAYFORM5 DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 It is important to note that matrices W are shared (i.e., separately applied) across all nodes. This can be viewed as 1-dimensional convolution with kernel and stride sizes of 1. Weight sharing across nodes is a key aspect of MPNNs, where node dependencies are learned in an order-invariant manner. Notations: We define the following notations, used throughout the paper. DISPLAYFORM0 be the set of data samples with inputs x ∈ X and outputs y ∈ Y. Inputs x are a (possibly ordered) set of S components {x 1, x 2, ..., x S}, and outputs y are a set of L labels {y 1, y 2, ..., y L}. MLC involves predicting the set of binary labels {y 1, y 2, ..., y L}, y i ∈ {0, 1} given input x. Input features are represented as embedded vectors {c DISPLAYFORM1, where d is the embedding size, δ is the vocabulary size, and t represents the 'state' of the embedding after t updates. Similarly, labels are represented as an embedded vectors {h DISPLAYFORM2 where L is the number of labels, and t represents the 'state' of the embedding after t updates. In MLC, each output label is determined by a joint probability of other labels and the input features. Our goal is to achieve the performance of explicit joint probability methods such as PCCs (Eqs. 22 and 23), at the test speed of BR methods (Eq. 21).We introduce Message Passing Encoder-Decoder (MPED) networks, where we formulate MLC using an encoder-decoder architecture. In MPED Networks, input components are represented as nodes in encoder graph G ENC using embedding vectors {c t 1:S}, and labels are represented as nodes in decoder graph G DEC using embedding vectors {h t 1:L}. MPED networks use three MPNN modules with attention to pass messages within G ENC, from G ENC to G DEC, and within G DEC to model the joint prediction of labels. The first module, MPNN xx, is used to update input component nodes {c t 1:S} by passing messages within the encoder (between input nodes). The second module, MPNN xy, is used to update output label nodes {h t 1:L} by passing messages from the encoder to decoder (from input nodes {c t 1:S} to output nodes {h t 1:L}). The third module, MPNN yy, is used to update output label nodes {h t 1:L} by passing messages within the decoder (between label nodes). Once messages have been passed to update input and label nodes, a readout function R is then used on the label nodes to make a binary classification prediction for each label, {ŷ 1,ŷ 2, ...,ŷ L}. An overview of our model is shown in Fig. 1. COMPONENT MESSAGE PASSING For a particular input x, we first assume that the input features {x 1:S} are nodes on a graph, we call G ENC. G ENC = (V, E), V = {x 1:S}, and E includes all undirected pairwise edges connecting node c i and node c j. MPNN xx, parameterized by W xx, is used to pass messages between the input embeddings in order to update their states. x i can be any component of a particular input (e.g. words in a sentence, patches of an image, nodes of a known graph, or tabular features).Nodes on G ENC are represented as embedding vectors {c : DISPLAYFORM0 DISPLAYFORM1 If there exists a known G ENC graph, message m t i for node i is computed using its neighboring nodes j ∈ N (i), where the neighbors N (i) are defined by the graph. If there is no known graph, we assume a fully connected G ENC graph, which means N (i) = {j = i}. Inputs with a sequential ordering can be modelled as a fully connected graph using positional embeddings BID9. Similar to the input components in the encoder, we assume that the labels {y 1:L} are nodes on a decoder graph called G DEC. Nodes on G DEC are represented as embedding vectors {h DISPLAYFORM0 where the initial states {h 0 1:L} are obtained using label embedding matrix W y. The decoder MPNNs update the label embeddings {h t 1:L} by passing messages from the encoder to the decoder, and then pass messages within the decoder. MPNN xy, is used to pass messages from input embeddings {c T 1:S} to label embeddings, and then MPNN yy is used to pass messages between label embeddings. In order to update the label nodes given a particular input x, the decoder uses MPNN xy, parameterized by W xy, to pass messages from input x to labels y. At the equation level, this module is identical to MPNN xx except that it updates the i th label node's embedding h i using the embeddings of all the components of an input. That is, we update each h t i by using a weighted sum of all input embeddings {c T 1:S}, in which the weights represent how important an input component is to the i th label node and the weights are learned via attention. Messages are only passed from the encoder nodes to the decoder nodes, and not vice versa (i.e. encoder to decoder message passing is directed). DISPLAYFORM0 DISPLAYFORM1 The key advantage of input-to-label message passing with attention is that each label node can attend to different input nodes (e.g. different words in the sentence). At this point, the decoder can make an independent prediction for each label conditioned on x. However, in order to make more accurate predictions, we model interactions between the label nodes {h t 1:L} using message passing and update them accordingly. To do this we use a a third message passing module, MPNN yy. At the equation level, this layer is identical to MPNN xx except that it replaces the input embeddings with label embeddings. In other words, label embedding h t i is updated by a weighted combination through attention of all its neighbor label nodes {h DISPLAYFORM0 To update each label embedding h : DISPLAYFORM1 DISPLAYFORM2 If there exists a known G DEC graph, message m t i for node i is computed using its neighboring nodes j ∈ N (i), where the neighbors N (i) are defined by the graph. If there is no known G DEC graph, we assume a fully connected graph, which means N (i) = {j = i}.In our implementation, the label embeddings are updated by MPNN xy and MPNN xx for T time steps to produce {h DISPLAYFORM3 The last module of the decoder predicts each label DISPLAYFORM0 is the learned output vector for label i. The calculated vector of size L × 1 is then fed through an element-wise sigmoid function to produce probabilities of each label being positive: DISPLAYFORM1 In MPED networks we use binary the mean cross entropy on the individual label predictions to train the model. p(y i |{y j =i}, c T 1:S; W) is approximated in MPED networks by jointly representing {y 1:L} using message passing from {c T 1:S} and from the embeddings of all neighboring labels {y j∈N (i) }. Multi-head Attention In order to allow a particular node to attend to multiple other nodes (or multiple groups of nodes) at once, MPED uses multiple attention heads. Inspired by BID8, we use K independent attention heads for each W · matrix during the message computation, where each matrix column W ·,k j is of dimension d/K. The generated representations are concatenated (denoted by) and linearly transformed by matrix W z ∈ R d×d. Multi-head attention changes message passing function M atn, but update function U mlp stays the same. DISPLAYFORM0 DISPLAYFORM1 Graph Time Steps To learn more complex relations among nodes, we compute T time steps of embedding updates. This is essentially a stack of T MPNN layers. Matrices BID10. DISPLAYFORM2 Speed. In MPED models, the joint probability of labels isn't explicitly estimated using the chain rule. This enables making predictions in parallel and decreases test time drastically, especially when the number of labels is large. We model the joint probability implicitly using the MPED decoder, at the benefit of a substantial speedup. Time complexities of different types of models are compared in Handling dense label predictions. Motivated by the drawbacks of autoregressive models for MLC (Section 5.6), the proposed MPED model removes the dependencies on a chosen label ordering and beam search. This is particularly beneficial when the number of positive output labels is large (i.e. dense). MPED networks predict the output set of labels all at once, which is made possible by the fact that inference doesn't use a probabilistic chain, but there is still a representation of label dependencies via label to label attention. As an additional benefit, as noted by, it may be useful to maintain'soft' predictions for each label in MLC. This is a major drawback of the PCC models which make'hard' predictions of the positive labels, defaulting all other labels to 0. DISPLAYFORM0 Flexibility Many input or output types are instances where the relational structure is not made explicit, and must be inferred or assumed (e.g., text corpora, or MLC labels) BID9. MPED networks allow for greater flexibility of input structures (known structure such as sequence or graph, or unknown such as tabular), or output structures (e.g., known graph vs unknown structure).Interpretability. One advantage of MPED models is that interpretability is "built in" via neural attention. Specifically, we can visualize 3 aspects: input-to-input attention (input dependencies), input-to-label attention (input/label dependencies), and label-to-label attention (label dependencies). Structured Output Predictions The use of graph attention in MPED models is closely connected to the literature of structured output prediction for MLC. BID12 used conditional random fields BID13 ) to model dependencies among labels and features for MLC by learning a distribution over pairs of labels to input features. In another research direction, recently proposed SPENs (structured prediction energy network BID14) and Deep Value Networks BID15 tackled MLC by optimizing different variants of structured loss formulations. In contrast to SPEN and related methods which use an iterative refinement of the output label predictions, our method is a simpler feed forward block to make predictions in one step, yet still models dependencies through attention mechanisms on embeddings. However, we plan to expand MPED models by adding a structured loss formulation. Graph Neural Networks (GNNs) Passing embedding messages from node to neighbor nodes connects to a large body of literature on graph neural networks BID9 ) and embedding models for structures BID16. The key idea is that instead of conducting probabilistic operations (e.g., product or re-normalization), the proposed models perform nonlinear function mappings in each step to learn feature representations of structured components. Neural message passing networks BID3, graph attention networks BID6 and neural relation models BID17 follow similar ideas to pass the embedding from node to neighbor nodes or neighbor edges. There have been many recent works extending the basic GNN framework to update nodes using various message passing, update, and readout functions BID18 BID19 BID20 BID21 BID3 BID17 BID22 BID23. We refer the readers to BID9 ) for a survey. However, none of these have used GNNs for MLC. In the appendix, we have added the details of training/hyperparameters (5.2), datasets (5.1), evaluation metrics (5.3), and explained how we selected baseline models from previous work (5.8). We explain our own MPED variations in 5.7 and previous work baselines in section 5.8.In short, we compare MPED to MPED Prior G DEC (by using a known label graph), MPED Edgeless G DEC (by removing label-to-label message passing), MPED Autoregressive (by predicting labels sequentially), MP BR (binary relevance MLP output used on the mean of MPNN xx embeddings), and the baselines reported in related works. We compare our models on seven real world datasets, which vary in the number of samples, number of labels, input type (sequential, tabular, graph), and output type (unknown, known label graph), as summarized in TAB8.Across all datasets, MPED outperforms or achieves similar as the baseline models. Most importantly, we show that autoregressive models are not crucial in MLC for most metrics, and non-autoregressive models in a significant speedup at test time. Table 2 shows the performance of different models across the 7 datasets. Significance across models is shown in Appendix TAB9. For subset accuracy (ACC), autoregressive models perform the best, but at a small margin of increase. However, autoregressive models that predict only positive labels are targeted at maximizing subset accuracy, but they perform poorly on other metrics. For all other metrics, autoregressive models are not essential. One important observation is that for most datasets, MPED outperforms the autoregressive models in both miF1 (frequent labels) and more importantly, maF1 (rare labels). Since maF1 favors models which can predict the rare labels, this shows that autoregressive models with beam search often make the wrong predictions on the rare labels (which are ordered last in the sequence during training). MPED is a solid choice across all metrics as it comes the closest to subset accuracy as the autoregressive models, but also performs well in other metrics. While MPED does not explicitly model label dependencies as autoregressive or structured prediction models do, it seems as though the attention weights do learn some dependencies among labels (Visualizations TAB8). This is indicated by the fact that MPED, which uses label-to-label attention, mostly outperforms the ones which don't, indicating that it is learning label dependencies. Table 2 shows 3 time step models , but a comparison of time steps is shown in Figure 2.Speed Table 2 shows the per epoch train and test times for each model. All models are trained and tested on the same GPU using a batch size of 32. At test time, since the autoregressive model cannot be parallelized, MPED and other non-autoregressive models are significantly faster. During training, the autoregressive model can be parallelized because the true labels are fed as the previous label. Since the autoregressive models only predict the ρ positive labels, they can be faster at training time, whereas the MPED model is predicting the probability for all labels. MPED in a mean of 1.7x and 5.0x training and testing speedups, respectively, over Seq2Seq autoregressive models. Interpretability We present visualizations of the input-to-label and label-to-label attention weights (averaged across the 4 attention heads) in the Appendix. In the visualizations, we show the positive Table 2: Results. Across all 7 datasets, MPED produces similar or better average metric scores to baseline models. MPED in a mean of 1.7x and 5.0x training and testing speedups, respectively, over the previous state-of-the-art probabilistic MLC method, RNN Seq2Seq. Speedups over RNN Seq2Seq model are shown in minutes per epoch in parentheses for the MPED model. Bold numbers show the best performing method(s). labels only, and the darker lines show higher attention weights to the corresponding label or word. The attention weights clearly learn certain relationships between input-label pairs as well as the label-label pairs, which is all done in an unsupervised manner. In future work, we plan to add a structured prediction loss function which will likely improve the attention mechanisms and the model's ability to estimate the joint probability. In this work we present Message Passing Encoder-Decoder (MPED) Networks which achieve a significant speedup at close to the same performance as autoregressive models for MLC. We open a new avenue of using neural message passing to model label dependencies in MLC tasks. In addition, we show that our method is able to handle various input data types (sequence, tabular, graph), as well various output label structures (known vs unknown). One of our future extensions is to adapt the current model to predict more dynamic outputs. BID1 BID24 We test our method against baseline methods on seven different multi-label sequence classification datasets. The datasets are summarized in TAB8. We use Reuters-21578, Bibtex BID30, which is side effects of drug molecules. As shown in the table, each dataset has a varying number of samples, number of labels, positive labels per sample, and samples per label. For BibTex and Delicious, we use 10% of the provided training set for validation. For the TFBS dataset, we use 1 layer of convolution at the first layer to extract "words" from the DNA characters (A,C,G,T), as commonly done in deep learning models for DNA.For datasets which have sequential ordering of the input components (Reuters, RCV1), we add a positional encoding to the word embedding as used in BID8 (sine and cosine functions of different frequencies) to encode the location of each word in the sentence. For datasets with no ordering or graph stucture (Bibtex, Delicious, Bookmarks, which use bag-of-word input representations) we do not use positional encodings. For inputs with an explicit graph representation (SIDER), we use the known graph structer. We validate our model on seven MLC datasets. These datasets cover a wide spectrum of input data types, including: raw English text (sequential form), bag-of-words (tabular form), and drug molecules (graph form).For all 6 datasets except SIDER, we use the same MPED model with T =3 time steps, d = 512, and K=4 attention heads. Since SIDER is significantly smaller, we use T =1 time step, d = 64, and K=4 attention heads. We trained our models on an NVIDIA TITAN X Pascal with a batch size of 32. We used Adam BID31 with betas= (0.9, 0.999), eps=1e-08, and a learning rate of 0.0002 for each dataset. We used dropout of p = 0.2 for all models. The MPED models also use layer normalization BID32 around each of the attention and feedforward layers. The non-autoregressive models are trained with binary cross-entropy on each label and the autoregressive models are trained with cross entropy across all possible labels at each position. Multi-label classification methods can be evaluated with many different metrics which each evaluate different strengths or weaknesses. We use the same 5 evaluation metrics from.All of our autoregressive models predict only the positive labels before outputting a stop signal. This is a special case of PCC models (explained as PCC+ in section 5.4), which have been shown to outperform the binary prediction of each label in terms of performance and speed. These models use beam search at inference time with a beam size of 5. For the non-autoregressive models, to convert the labels to {0, 1} we chose the best threshold on the validation set from the same set of thresholds used in BID14.Example-based measures are defined by comparing the target vector y to the prediction vectorŷ. Subset Accuracy (ACC) requires an exact match of the predicted labels and the true labels: ACC(y,ŷ) = I[y =ŷ].Hamming Accuracy (HA) evaluates how many labels are correctly predicted inŷ: DISPLAYFORM0 Example-based F1 (ebF1) measures the ratio of correctly predicted labels to the sum of the total true and predicted labels: DISPLAYFORM1 Label-based measures treat each label yj as a separate two-class prediction problem, and compute the number of true positives (tpj), false positives (f pj), and false negatives (f nj) for a label. Macro-averaged F1 (maF1) measures the label-based F1 averaged over all labels: DISPLAYFORM2 measures the label-based F1 averaged over each sample: DISPLAYFORM3. High maF1 scores usually indicate high performance on less frequent labels. High miF1 scores usually indicate high performance on more frequent labels. MLC has a rich history in text (McCallum; BID34, images BID0 BID35, bioinformatics BID0 BID35, and many other domains. MLC methods can roughly be broken into several groups, which are explained as follows. Label powerset models (LP) BID36 BID37, classify each input into one label combination from the set of all possible combinations Y = {{1}, {2},..., {1, 2, ..., L}}. LP explicitly models the joint distribution by predicting the one subset of all positive labels. Since the label set Y grows exponentially in the number of total labels (2 L), classifying each possible label set is intractable for a modest L. In addition, even in small L tasks, LP suffers from the "subset scarcity problem" where only a small amount of the label subsets are seen during training, leading to bad generalization. Binary relevance (BR) methods predict each label separately as a logistic regression classfier for each label BID38 BID39. The naïve approach to BR prediction is to predict all labels independently of one another, assuming no dependencies among labels. That is, BR uses the following conditional probability parameterized by learned weights W: DISPLAYFORM0 Probabilistic classifier chain (PCC) methods BID40 BID1 ) are autoregressive models that estimate the true joint probability of output labels given the input by using the chain rule, predicting one label at a time: DISPLAYFORM1 Two issues with PCC models are that inference is very slow if L is large, and the errors propagate as L increases BID41. To mitigate the problems with both LP and PCC methods, one solution is to only predict the true labels in the LP subset. In other words, only predicting the positive labels (total of ρ for a particular sample) and ignoring all other labels, which we call PCC+. Similar to PCC, the joint probability of PCC+ can be computed as product of conditional probabilities, but unlike PCC, only ρ < L terms are predicted as positive: DISPLAYFORM2 This can be beneficial when the number of possible labels L is large, reducing the total number of prediction steps. However, in both PCC and PCC+, inference is done using beam search, which is a costly dynamic programming step to find the optimal prediction. MPED methods approximate the following factored formulation, where N (Yi) denotes the neighboring nodes of Yi. DISPLAYFORM3 In machine translation (MT), sequence-to-sequence (Seq2Seq) models have proven to be the superior method, where an encoder RNN reads the source language sentence into an encoder hidden state, and a decoder RNN Figure 2: Average Across Metrics for T =1, T =2, and T =3 GDEC time steps. In these experiments, the encoder GENC is processed with a fixed T =3 time steps, and the decoder time steps are varied. We do not compare time steps for the SIDER dataset since it is too small, and we only evaluate using T =1.translates the hidden state into a target sentence, predicting each word autoregressively BID42. BID7 improved this model by introducing "neural attention" which allows the decoder RNN to "attend" to every encoder word at each step of the autoregressive translation. Recently, showed that, across several metrics, state-of-the-art MLC could be achieved by using a recurrent neural network (RNN) based encoder-to-decoder framework for Equation 23 (PCC+). They use a Seq2Seq RNN model (Seq2Seq Autoregressive) which uses one RNN to encode x, and a second RNN to predict each positive label sequentially, until it predicts a'stop' signal. This type of model seeks to maximize the'subset accuracy', or correctly predict every label as its exact 0/1 value. BID8 eliminated the need for the recurrent network in MT by introducing the Transformer. Instead of using an RNN to model dependencies, the Transformer explicitly models pairwise dependencies among all of the features by using attention BID7 BID43 between signals. This speeds up training time because RNNs can't be fully parallelized but, the transformer still uses an autoregressive decoder. Autoregressive models have been proven effective for machine translation and MLC BID42 BID7. However, predictions must be made sequentially, eliminating parallelization. Also, beam search is typically used at test time to find optimal predictions. But beam search is limited by the time cost of large beams sizes, making it difficult to optimally predict many output labels BID44.In addition to speed constraints, beam search for autoregressive inference introduces a second drawback: initial wrong predictions will propagate when using a modest beam size (e.g. most models use a beam size of 5). This can lead to significant decreases in performance when the number of positive labels is large. For example, the Delicious dataset has a median of 19 positive labels per sample, and it can be very difficult to correctly predict the labels at the end of the prediction chain. Autoregressive models are well suited for machine translation because these models mimic the sequential decoding process of real translation. However, for MLC, the output labels have no intrinsic ordering. While the joint probability of the output labels is independent of the label ordering via autoregressive based inference, the chosen ordering can make a difference in practice BID45. Some ordering of labels must be used during training, and this chosen ordering can lead to unstable predictions at test time. Our non-autoregressive version connects to BID46 who removed the autoregressive decoder in MT with the Non-Autoregressive Transformer. In this model, the encoder makes a proxy prediction, called "fertilities", which are used by the decoder to predict all translated words at once. The difference between their model and ours is that we have a constant label at each position, so we don't need to marginalize over all possible labels at each position. In the full MPED model, we use 3 encoder time steps and 3 decoder time steps with node to node attention in both the encoder and decoder graphs, and K=4 attention heads.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Message Passing Encoder-Decode networks for a fast and accurate way of modelling label dependencies for multi-label classification.

Represent the TLDR for finding the corresponding scientific article:

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.