Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Graph neural networks have recently achieved great successes in predicting quantum mechanical properties of molecules. These models represent a molecule as a graph using only the distance between atoms (nodes) and not the spatial direction from one atom to another. However, directional information plays a central role in empirical potentials for molecules, e.g. in angular potentials. To alleviate this limitation we propose directional message passing, in which we embed the messages passed between atoms instead of the atoms themselves. Each message is associated with a direction in coordinate space. These directional message embeddings are rotationally equivariant since the associated directions rotate with the molecule. We propose a message passing scheme analogous to belief propagation, which uses the directional information by transforming messages based on the angle between them. Additionally, we use spherical Bessel functions to construct a theoretically well-founded, orthogonal radial basis that achieves better performance than the currently prevalent Gaussian radial basis functions while using more than 4x fewer parameters. We leverage these innovations to construct the directional message passing neural network (DimeNet). DimeNet outperforms previous GNNs on average by 77% on MD17 and by 41% on QM9. In recent years scientists have started leveraging machine learning to reduce the computation time required for predicting molecular properties from a matter of hours and days to mere milliseconds. With the advent of graph neural networks (GNNs) this approach has recently experienced a small revolution, since they do not require any form of manual feature engineering and significantly outperform previous models. GNNs model the complex interactions between atoms by embedding each atom in a high-dimensional space and updating these embeddings by passing messages between atoms. By predicting the potential energy these models effectively learn an empirical potential function. Classically, these functions have been modeled as the sum of four parts: where E bonds models the dependency on bond lengths, E angle on the angles between bonds, E torsion on bond rotations, i.e. the dihedral angle between two planes defined by pairs of bonds, and E non-bonded models interactions between unconnected atoms, e.g. via electrostatic or van der Waals interactions. The update messages in GNNs, however, only depend on the previous atom embeddings and the pairwise distances between atoms -not on directional information such as bond angles and rotations. Thus, GNNs lack the second and third terms of this equation and can only model them via complex higher-order interactions of messages. Extending GNNs to model them directly is not straightforward since GNNs solely rely on pairwise distances, which ensures their invariance to translation, rotation, and inversion of the molecule, which are important physical requirements. In this paper, we propose to resolve this restriction by using embeddings associated with the directions to neighboring atoms, i.e. by embedding atoms as a set of messages. These directional message embeddings are equivariant with respect to the above transformations since the directions move with the molecule. Hence, they preserve the relative directional information between neighboring atoms. We propose to let message embeddings interact based on the distance between atoms and the angle between directions. Both distances and angles are invariant to translation, rotation, and inversion of the molecule, as required. Additionally, we show that the distance and angle can be jointly represented in a principled and effective manner by using spherical Bessel functions and spherical harmonics. We leverage these innovations to construct the directional message passing neural network (DimeNet). DimeNet can learn both molecular properties and atomic forces. It is twice continuously differentiable and solely based on the atom types and coordinates, which are essential properties for performing molecular dynamics simulations. DimeNet outperforms previous GNNs on average by 76 % on MD17 and by 31 % on QM9. Our paper's main contributions are: 1. Directional message passing, which allows GNNs to incorporate directional information by connecting recent advances in the fields of equivariance and graph neural networks as well as ideas from belief propagation and empirical potential functions such as Eq. 1. 2. Theoretically principled orthogonal basis representations based on spherical Bessel functions and spherical harmonics. Bessel functions achieve better performance than Gaussian radial basis functions while reducing the radial basis dimensionality by 4x or more. 3. The Directional Message Passing Neural Network (DimeNet): A novel GNN that leverages these innovations to set the new state of the art for molecular predictions and is suitable both for predicting molecular properties and for molecular dynamics simulations. ML for molecules. The classical way of using machine learning for predicting molecular properties is combining an expressive, hand-crafted representation of the atomic neighborhood (Bartók et al., 2013) with Gaussian processes (Bartók et al., 2010; or neural networks . Recently, these methods have largely been superseded by graph neural networks, which do not require any hand-crafted features but learn representations solely based on the atom types and coordinates molecules . Our proposed message embeddings can also be interpreted as directed edge embeddings. (Undirected) edge embeddings have already been used in previous GNNs (Jørgensen et al., 2018;). However, these GNNs use both node and edge embeddings and do not leverage any directional information. Graph neural networks. GNNs were first proposed in the 90s and 00s . General GNNs have been largely inspired by their application to molecular graphs and have started to achieve breakthrough performance in various tasks at around the same time the molecular variants did (; ;). Some recent progress has been focused on GNNs that are more powerful than the 1-Weisfeiler-Lehman test of isomorphism . However, for molecular predictions these models are significantly outperformed by GNNs focused on molecules (see Sec. 7). Some recent GNNs have incorporated directional information by considering the change in local coordinate systems per atom . However, this approach breaks permutation invariance and is therefore only applicable to chain-like molecules (e.g. proteins). Equivariant neural networks. Group equivariance as a principle of modern machine learning was first proposed by. Following work has generalized this principle to spheres, molecules , volumetric data , and general manifolds . Equivariance with respect to continuous rotations has been achieved so far by switching back and forth between Fourier and coordinate space in each layer or by using a fully Fourier space model . The former introduces major computational overhead and the latter imposes significant constraints on model construction, such as the inability of using non-linearities. Our proposed solution does not suffer from either of those limitations. In recent years machine learning has been used to predict a wide variety of molecular properties, both low-level quantum mechanical properties such as potential energy, energy of the highest occupied molecular orbital (HOMO), and the dipole moment and high-level properties such as toxicity, permeability, and adverse drug reactions . In this work we will focus on scalar regression targets, i.e. targets t ∈ R. A molecule is uniquely defined by the atomic numbers z = {z 1, . . ., z N} and positions X = {x 1, . . ., x N}. Some models additionally use auxiliary information Θ such as bond types or electronegativity of the atoms. We do not include auxiliary features in this work since they are hand-engineered and non-essential. In summary, we define an ML model for molecular prediction with parameters θ via f θ: {X, z} → R. Symmetries and invariances. All molecular predictions must obey some basic laws of physics, either explicitly or implicitly. One important example of such are the fundamental symmetries of physics and their associated invariances. In principle, these invariances can be learned by any neural network via corresponding weight matrix symmetries . However, not explicitly incorporating them into the model introduces duplicate weights and increases training time and complexity. The most essential symmetries are translational and rotational invariance (follows from homogeneity and isotropy), permutation invariance (follows from the indistinguishability of particles), and symmetry under parity, i.e. under sign flips of single spatial coordinates. Molecular dynamics. Additional requirements arise when the model should be suitable for molecular dynamics (MD) simulations and predict the forces F i acting on each atom. The force field is a conservative vector field since it must satisfy conservation of energy (the necessity of which follows from homogeneity of time ). The easiest way of defining a conservative vector field is via the gradient of a potential function. We can leverage this fact by predicting a potential instead of the forces and then obtaining the forces via backpropagation to the atom coordinates, i.e. We can even directly incorporate the forces in the training loss and directly train a model for MD simulations : where the targett =Ê is the ground-truth energy (usually available as well),F are the ground-truth forces, and the hyperparameter ρ sets the forces' loss weight. For stable simulations F i must be continuously differentiable and the model f θ itself therefore twice continuously differentiable. We hence cannot use discontinuous transformations such as ReLU non-linearities. Furthermore, since the atom positions X can change arbitrarily we cannot use pre-computed auxiliary information Θ such as bond types. Graph neural networks. Graph neural networks treat the molecule as a graph, in which the nodes are atoms and edges are defined either via a predefined molecular graph or simply by connecting atoms that lie within a cutoff distance c. Each edge is associated with a pairwise distance between atoms d ij = x i − x j 2. GNNs implement all of the above physical invariances by construction since they only use pairwise distances and not the full atom coordinates. However, note that a predefined molecular graph or a step function-like cutoff cannot be used for MD simulations since this would introduce discontinuities in the energy landscape. GNNs represent each atom i via an atom embedding h i ∈ R H. The atom embeddings are updated in each layer by passing messages along the molecular edges. Messages are usually transformed based on an edge embedding e (ij) ∈ R He and summed over the atom's neighbors N i, i.e. the embeddings are updated in layer l via with the update function f update and the interaction function f int, which are both commonly implemented using neural networks. The edge embeddings e (l) (ij) usually only depend on the interatomic distances, but can also incorporate additional bond information or be recursively updated in each layer using the neighboring atom embeddings (Jørgensen et al., 2018). Directionality. In principle, the pairwise distance matrix contains the full geometrical information of the molecule. However, GNNs do not use the full distance matrix since this would mean passing messages globally between all pairs of atoms, which increases computational complexity and can lead to overfitting. Instead, they usually use a cutoff distance c, which means they cannot distinguish between certain molecules . E.g. at a cutoff of roughly 2 Å a regular GNN would not be able to distinguish between a hexagonal (e.g. Cyclohexane) and two triangular molecules (e.g. Cyclopropane) with the same bond lengths since the neighborhoods of each atom are exactly the same for both (see Appendix, Fig. 6). This problem can be solved by modeling the directions to neighboring atoms instead of just their distances. A principled way of doing so while staying invariant to a transformation group G (such as described in Sec. 3) is via group-equivariance with the group action in the input and output space ϕ X g and ϕ Y g. However, equivariant CNNs only achieve equivariance with respect to a discrete set of rotations . For a precise prediction of molecular properties we need continuous equivariance with respect to rotations, i.e. to the SO group. Directional embeddings. We solve this problem by noting that an atom by itself is rotationally invariant. This invariance is only broken by neighboring atoms that interact with it, i.e. those inside the cutoff c. Since each neighbor breaks up to one rotational invariance they also introduce additional degrees of freedom, which we need to represent in our model. We can do so by generating a separate embedding m ji for each atom i and neighbor j by applying the same learned filter in the direction of each neighboring atom (in contrast to equivariant CNNs, which apply filters in fixed, global directions). These directional embeddings are equivariant with respect to global rotations since the associated directions rotate with the molecule and hence conserve the relative directional information between neighbors. Representation via joint 2D basis. We use the directional information associated with each embedding by leveraging the angle α (kj,ji) = ∠x k x j x i when aggregating the neighboring embeddings m kj of m ji. We combine the angle with the interatomic distance d kj associated with the incoming message m kj and jointly represent both in a (kj,ji) SBF ∈ R NSHBF·NSRBF using a 2D representation based on spherical Bessel functions and spherical harmonics, as explained in Sec. 5. We empirically found that this basis representation provides a better inductive bias than the raw angle alone. Message embeddings. The directional embedding m ji associated with the atom pair ji can be thought of as a message being sent from atom j to atom i. Hence, in analogy to belief propagation, we embed each atom i using a set of incoming messages m ji, i.e. h i = j∈Ni m ji, and update the message m ji based on the incoming messages m kj . Hence, as illustrated in Fig. 1, we define the update function and aggregation scheme for message embeddings as where e (ji) RBF denotes the radial basis function representation of the interatomic distance d ji, which will be discussed in Sec. 5. We found this aggregation scheme to not only have a nice analogy to belief propagation, but also to empirically perform better than alternatives. Note that since f int now incorporates the angle between atom pairs, or bonds, we have enabled our model to directly learn the angular potential E angle, the second term in Eq. 1. Moreover, the message embeddings are essentially embeddings of atom pairs, as used by the provably more powerful GNNs based on higher-order Weisfeiler-Lehman tests of isomorphism. Our model can therefore provably distinguish molecules that a regular GNN cannot (e.g. the previous example of a hexagonal and two triangular molecules) . Representing distances and angles. For the interaction function f int in Eq. 4 we use a joint representation a (kj,ji) SBF of the angles α (kj,ji) between message embeddings and the interatomic distances d kj = x k − x j 2, as well as a representation e (ji) RBF of the distances d ji. Earlier works have used a set of Gaussian radial basis functions to represent interatomic distances, with tightly spaced means that are distributed e.g. uniformly or exponentially . Similar in spirit to the functional bases used by steerable CNNs we propose to use an orthogonal basis instead, which reduces redundancy and thus improves parameter efficiency. Furthermore, a basis chosen according to the properties of the modeled system can even provide a helpful inductive bias. We therefore derive a proper basis representation for quantum systems next. with the spherical Bessel functions of the first and second kind j l and y l and the spherical harmonics Y m l. As common in physics we only use the regular solutions, i.e. those that do not approach −∞ at the origin, and hence set b lm = 0. Recall that our first goal is to construct a joint 2D basis for d kj and α (kj,ji), i.e. a function that depends on d and a single angle α. To achieve this we set m = 0 and obtain. The boundary conditions are satisfied by setting k = z ln c, where z ln is the n-th root of the l-order Bessel function, which are precomputed numerically. Normalizing Ψ SBF inside the cutoff distance c yields the 2D spherical Fourier-Bessel basisã (kj,ji) SBF ∈ R NSHBF·NSRBF, which is illustrated in Fig. 2 and defined byã with n ∈ [1 . . N RBF]. Both of these bases are purely real-valued and orthogonal in the domain of interest. They furthermore enable us to bound the highest-frequency components by ω α ≤ NSHBF 2π, ω d kj ≤ NSRBF c, and ω dji ≤ NRBF c. This restriction is an effective way of regularizing the model and ensures that predictions are stable to small perturbations. We found N SRBF = 6 and N RBF = 16 radial basis functions to be more than sufficient. Note that N RBF is 4x lower than PhysNet's 64 and 20x lower than SchNet's 300 radial basis functions. and their first and second derivatives to go to 0 at the cutoff. We achieve this with the polynomial where p ∈ N 0. We did not find the model to be sensitive to different choices of envelope functions and choose p = 3. Note that using an envelope function causes the bases to lose their orthonormality, which we did not find to be a problem in practice. We furthermore fine-tune the Bessel wave numbers k n = nπ c used inẽ RBF ∈ R NRBF via backpropagation after initializing them to these values, which we found to give a small boost in prediction accuracy. The Directional Message Passing Neural Network's (DimeNet) design is based on a streamlined version of the PhysNet architecture , in which we have integrated directional message passing and spherical Fourier-Bessel representations. DimeNet generates predictions that are invariant to atom permutations and translation, rotation and inversion of the molecule. DimeNet is suitable both for the prediction of various molecular properties and for molecular dynamics (MD) simulations. It is twice continuously differentiable and able to learn and predict atomic forces via backpropagation, as described in Sec. 3. The predicted forces fulfill energy conservation by construction and are equivariant with respect to permutation and rotation. Model differentiability in combination with basis representations that have bounded maximum frequencies furthermore guarantees smooth predictions that are stable to small deformations. Fig. 4 gives an overview of the architecture. Embedding block. Atomic numbers are represented by learnable, randomly initialized atom type embeddings h i ∈ R F that are shared across molecules. The first layer generates message embeddings from these and the distance between atoms via where denotes concatenation and the weight matrix W and bias b are learnable. Interaction block. The embedding block is followed by multiple stacked interaction blocks. This block implements f int and f update of Eq. 4 as shown in Fig. 4. Note that the 2D representation a (kj,ji) SBF is first transformed into an N tensor -dimensional representation via a linear layer. The main purpose of this is to make the dimensionality of a (kj,ji) SBF independent of the subsequent bilinear layer, which uses a comparatively large N tensor × F × F -dimensional weight tensor. We have also experimented with using a bilinear layer for the radial basis representation, but found that the element-wise multiplication e RBF W m kj performs better, which suggests that the 2D representations require more complex transformations than radial information alone. The interaction block transforms each message embedding m ji using multiple residual blocks, which are inspired by ResNet and consist of two stacked dense layers and a skip connection. Output block. The message embeddings after each block (including the embedding block) are passed to an output block. The output block transforms each message embedding m ji using the radial basis e (ji) RBF, which ensures continuous differentiability and slightly improves performance. Afterwards the incoming messages are summed up per atom i to obtain h i = j m ji, which is then transformed using multiple dense layers to generate the atom-wise output t (l) i. These outputs are then summed up to obtain the final prediction t = i l t (l) i. Continuous differentiability. Multiple model choices were necessary to achieve twice continuous model differentiability. First, DimeNet uses the self-gated Swish activation function σ(x) = x · sigmoid(x) instead of a regular ReLU activation function. Second, we multiply the radial basis functionsẽ RBF (d) with an envelope function u(d) that has a root of multiplicity 3 at the cutoff c. Finally, DimeNet does not use any auxiliary data but relies on atom types and positions alone. Models. For hyperparameter choices and training setup see Appendix B. We use 6 state-of-the-art models for comparison: SchNet, PhysNet (whose we have generated ourselves using the reference implementation) , provably powerful graph networks (PPGN) , MEGNet-simple (the variant without auxiliary information) , Cormorant , and symmetrized gradient-domain machine learning (sGDML) . Note that sGDML cannot be used for QM9 since it can only be trained on a single molecule. We test DimeNet's performance for predicting molecular properties using the common QM9 benchmark . It consists of roughly 130 000 molecules in equilibrium with up to 9 heavy C, O, N, and F atoms. We use 110 000 molecules in the training, 10 000 in the validation and 13 885 in test set. We only use the atomization energy for U 0, U, H, and G, i.e. subtract the atomic reference energies, which are constant per atom type. In Table 1 we report the mean absolute error (MAE) of each target and the overall mean standardized MAE (std. MAE) and mean standardized logMAE (for details see Appendix C). We predict ∆ simply by taking. We use MD17 to test model performance in molecular dynamics simulations. The goal of this benchmark is predicting both the energy and atomic forces of eight small organic molecules, given the atom coordinates of the thermalized (i.e. non-equilibrium, slightly moving) system. The ground truth data is computed via molecular dynamics simulations using DFT. A separate model is trained for each molecule, with the goal of providing highly accurate individual predictions. This dataset is commonly used with 50 000 training and 10 000 validation and test samples. We found that DimeNet can match state-of-the-art performance in this setup. E.g. for Benzene, depending on the force weight ρ, DimeNet achieves 0.035 kcal mol −1 MAE for the energy or 0.07 kcal mol −1 and 0.17 kcal mol for energy and forces, matching the reported by and. However, this accuracy is two orders of magnitude below the DFT calculation's accuracy (approx. 2.3 kcal mol −1 for energy ), so any remaining difference to real-world data is almost exclusively due to errors in the DFT simulation. Truly reaching better accuracy can therefore only be achieved with more precise ground-truth data, which requires far more expensive methods (e.g. CCSD(T)) and thus ML models that are more sample-efficient . We therefore instead test our model on the harder task of using only 1000 training samples. As shown in Table 2 DimeNet outperforms SchNet by a large margin and performs roughly on par with sGDML. However, sGDML uses hand-engineered descriptors that provide a strong advantage for small datasets, can only be trained on a single molecule (a fixed set of atoms), and does not scale well with the number of atoms or training samples. Ablation studies. To test whether directional message passing and the Fourier-Bessel basis are the actual reason for DimeNet's improved performance, we ablate them individually and compare the mean standardized MAE and logMAE for multi-task learning on QM9. Table 3 shows that both of our contributions have a significant impact on the model's performance. Using 64 Gaussian RBFs instead of 16 and 6 Bessel basis functions to represent d ji and d kj increases the error by 10 %, which shows that this basis does not only reduce the number of parameters but additionally provides a helpful inductive bias. DimeNet's error increases by around 26 % when we ignore the angles between messages by setting N SHBF = 1, showing that directly incorporating directional information does indeed improve performance. Using node embeddings instead of message embeddings (and hence also ignoring directional information) has the largest impact and increases MAE by 68 %, at which point DimeNet performs worse than SchNet. Furthermore, Fig. 5 shows that the filters exhibit a structurally meaningful dependence on both the distance and angle. For example, some of these filters are clearly being activated by benzene rings (120 • angle, 1.39 Å distance). This further demonstrates that the model learns to leverage directional information. In this work we have introduced directional message passing, a more powerful and expressive interaction scheme for molecular predictions. Directional message passing enables graph neural networks to leverage directional information in addition to the interatomic distances that are used by normal GNNs. We have shown that interatomic distances can be represented in a principled and effective manner using spherical Bessel functions. We have furthermore shown that this representation can be extended to directional information by leveraging 2D spherical Fourier-Bessel basis functions. We have leveraged these innovations to construct DimeNet, a GNN suitable both for predicting molecular properties and for use in molecular dynamics simulations. We have demonstrated DimeNet's performance on QM9 and MD17 and shown that our contributions are the essential ingredients that enable DimeNet's state-of-the-art performance. DimeNet directly models the first two terms in Eq. 1, which are known as the important "hard" degrees of freedom in molecules . Future work should aim at also incorporating the third and fourth terms of this equation. This could improve predictions even further and enable the application to molecules much larger than those used in common benchmarks like QM9. Figure 6: A standard non-directional GNN cannot distinguish between a hexagonal (left) and two triangular molecules (right) with the same bond lengths, since the neighborhood of each atom is exactly the same. An example of this would be Cyclohexane and two Cyclopropane molecules with slightly stretched bonds, when the GNN either uses the molecular graph or a cutoff distance of c ≤ 2.5 Å. Directional message passing solves this problem by considering the direction of each bond. The model architecture and hyperparameters were optimized using the QM9 validation set. We use 6 stacked interaction blocks and embeddings of size F = 128 throughout the model. For the basis functions we choose N SHBF = 7, N SRBF = 6, and N RBF = 16 and N tensor = 12 for the weight tensor in the interaction block. We did not find the model to be very sensitive to these values as long as they were chosen large enough (i.e. at least 8). To illustrate the filters learned by DimeNet we separate the spatial dependency in the interaction function f int via f int (m, d ji, d kj, α (kj,ji) ) = n [σ(W m + b)] n f filter1,n (d ji)f filter2,n (d kj, α (kj,ji) ). The filters f filter1,n: R + → R and f filter2,n: R + × [0, 2π] → R F are given by where W RBF, W SBF, and W are learned weight matrices/tensors, e RBF (d) is the radial basis representation, and a SBF (d, α) is the 2D spherical Fourier-Bessel representation. Fig. 5 shows how the first 15 elements of f filter2,n (d, α) vary with d and α when choosing the tensor slice n = 1 (with α = 0 at the top of the figure). E MULTI-TARGET

Directional message passing incorporates spatial directional information to improve graph neural networks.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. The agent needs to learn a latent representation together with a control policy to perform the task. Fitting a high-capacity encoder using a scarce reward signal is not only extremely sample inefficient, but also prone to suboptimal convergence. Two ways to improve sample efficiency are to learn a good feature representation and use off-policy algorithms. We dissect various approaches of learning good latent features, and conclude that the image reconstruction loss is the essential ingredient that enables efficient and stable representation learning in image-based RL. Following these findings, we devise an off-policy actor-critic algorithm with an auxiliary decoder that trains end-to-end and matches state-of-the-art performance across both model-free and model-based algorithms on many challenging control tasks. We release our code to encourage future research on image-based RL. Cameras are a convenient and inexpensive way to acquire state information, especially in complex, unstructured environments, where effective control requires access to the proprioceptive state of the underlying dynamics. Thus, having effective RL approaches that can utilize pixels as input would potentially enable solutions for a wide range of real world problems. The challenge is to efficiently learn a mapping from pixels to an appropriate representation for control using only a sparse reward signal. Although deep convolutional encoders can learn good representations (upon which a policy can be trained), they require large amounts of training data. As existing reinforcement learning approaches already have poor sample complexity, this makes direct use of pixel-based inputs prohibitively slow. For example, model-free methods on Atari and DeepMind Control (DMC) take tens of millions of steps , which is impractical in many applications, especially robotics. A natural solution is to add an auxiliary task with an unsupervised objective to improve sample efficiency. The simplest option is an autoencoder with a pixel reconstruction objective. Prior work has attempted to learn state representations from pixels with autoencoders, utilizing a two-step training procedure, where the representation is first trained via the autoencoder, and then either with a policy learned on top of the fixed representation (; ; b; ;), or with planning . This allows for additional stability in optimization by circumventing dueling training objectives but leads to suboptimal policies. Other work utilizes end-to-end model-free learning with an auxiliary reconstruction signal in an on-policy manner . We revisit the concept of adding an autoencoder to model-free RL approaches, but with a focus on off-policy algorithms. We perform a sequence of careful experiments to understand why previous approaches did not work well. We found that a pixel reconstruction loss is vital for learning a good representation, specifically when trained end-to-end. Based on these findings, we propose a simple autoencoder-based off-policy method that can be trained end-to-end. Our method is the first modelfree off-policy algorithm to successfully train simultaneously both the latent state representation and policy in a stable and sample-efficient manner. used in our experiments. Each task offers an unique set of challenges, including complex dynamics, sparse rewards, hard exploration, and more. Refer to Appendix A for more information. Of course, some recent state-of-the-art model-based RL methods have demonstrated superior sample efficiency to leading model-free approaches on pixel tasks from . But we find that our model-free, off-policy, autoencoder-based approach is able to match their performance, closing the gap between model-based and model-free approaches in image-based RL, despite being a far simpler method that does not require a world model. This paper makes three main contributions: (i) a demonstration that adding a simple auxiliary reconstruction loss to a model-free off-policy RL algorithm achieves comparable to state-of-the-art model-based methods on the suite of continuous control tasks from; (ii) an understanding of the issues involved with combining autoencoders with model-free RL in the off-policy setting that guides our algorithm; and (iii) an open-source PyTorch implementation of our simple method for researchers and practitioners to use as a strong baseline that may easily be built upon. Efficient learning from high-dimensional pixel observations has been a problem of paramount importance for model-free RL. While some impressive progress has been made applying model-free RL to domains with simple dynamics and discrete action spaces , attempts to scale these approaches to complex continuous control environments have largely been unsuccessful, both in simulation and the real world. A glaring issue is that the RL signal is much sparser than in supervised learning, which leads to sample inefficiency, and higher dimensional observation spaces such as pixels worsens this problem. One approach to alleviate this problem is by training with auxiliary losses. Early work explores using deep autoencoders to learn feature spaces in visual reinforcement learning, crucially propose to recompute features for all collected experiences after each update of the autoencoder, rendering this approach impractical to scale to more complicated domains. Moreover, this method has been only demonstrated on toy problems. apply deep autoencoder pretraining to real world robots that does not require iterative re-training, improving upon computational complexity of earlier methods. However, in this work the linear policy is trained separately from the autoencoder, which we find to not perform as well as end-to-end methods. use auxiliary losses in Atari that incorporate forward and inverse dynamics with A3C, an on-policy algorithm. They recommend a multi-task setting and learning dynamics and reward to find a good representation, which relies on the assumption that the dynamics in the task are easy to learn and useful for learning a good policy. propose to use unsupervised auxiliary tasks, both observation-based and reward-based based off of real world inductive priors, and show improvements in Atari, again in the on-policy regime, which is much more stable for learning. Unfortunately, this work also relies on inductive biases by designing internal rewards to learn a good representation which is hard to scale to the real world problems. Higgins et al. (2017b); use a beta variational autoencoder (β-VAE) (; a) and attempt to extend unsupervised representation pretraining to the off-policy setting, but find it hard to perform end-to-end training, thus receding to the iterative retraining procedure . There has been more success in using model-based methods on images, such as;. These methods use a world model approach, learning a representation space using a latent dynamics loss and pixel decoder loss to ground on the original observation space. These model-based reinforcement learning methods often show improved sample efficiency, but with the additional complexity of balancing various auxiliary losses, such as a dynamics loss, reward loss, and decoder loss in addition to the original policy and value optimiza-tions. These proposed methods are correspondingly brittle to hyperparameter settings, and difficult to reproduce, as they balance multiple training objectives. To close the gap between model-based and model-free image-based RL in terms of sample efficiency and sidestep the issues of model learning, our goal is to train a model-free off-policy algorithm with auxiliary reconstruction loss in a stable manner. A fully observable Markov decision process (MDP) is described by tuple S, A, P, R, γ, where S is the state space, A is the action space, P (s t+1 |s t, a t) is the probability distribution over transitions, R(s t, a t, s t+1) is the reward function, and γ is the discount factor . An agent starts in a initial state s 1 sampled from a fixed distribution p(s 1), then at each timestep t it takes an action a t ∈ A from a state s t ∈ S and moves to a next state s t+1 ∼ P (·|s t, a t). After each action the agent receives a reward r t = R(s t, a t, s t+1). We consider episodic environments with the length fixed to T. The goal of standard RL is to learn a policy π(a t |s t) that can maximize the agent's expected cumulative reward, where ρ π is a state-action marginal distribution induced by the policy π(a t |s t) and transition distribution P (s t+1 |s t, a t). An important modification auguments this objective with an entropy term H(π(·|s t)) to encourage exploration and robustness to noise. The ing maximum entropy objective is then defined as: where α is a temperature parameter that balances between optimizing for the reward and for the stochasticity of the policy. We build on Soft Actor-Critic (SAC) , an off-policy actor-critic method that uses the maximum entropy framework to derive soft policy iteration. At each iteration SAC performs a soft policy evaluation step and a soft policy improvement step. The soft policy evaluation step fits a parametric soft Q-function Q(s t, a t) (critic) by minimizing the soft Bellman residual: where D is the replay buffer, andQ is the target soft Q-function parametrized by a weight vector obtained using the exponentially moving average of the soft Q-function weights to stabilize training. The soft policy improvement step then attempts to learn a parametric policy π(a t |s t) (actor) by directly minimizing the KL divergence between the policy and a Boltzmann distribution induced by the current soft Q-function, producing the following objective: The policy π(a t |s t) is parametrized as a diagonal Gaussian to handle continuous action spaces. When learning from raw images, we deal with the problem of partial observability, which is formalized by a partially observable MDP (POMDP). In this setting, instead of getting a low-dimensional state s t ∈ S at time t, the agent receives a high-dimensional observation o t ∈ O, which is a rendering of potentially incomplete view of the corresponding state s t of the environment . This complicates applying RL as the agent now needs to also learn a compact latent representation to infer the state. Fitting a high-capacity encoder using only a scarce reward signal is sample inefficient and prone to suboptimal convergence. Following prior work we explore unsupervised pretraining via an image-based autoencoder. In practice, the autoencoder is represented as a convolutional encoder f enc that maps an image observation o t to a low-dimensional latent vector z t, and a deconvolutional decoder f dec that reconstructs z t back to the original image o t. The optimization is done by minimizing the standard reconstruction objective: where Or in the case of β-VAE (; a), where the variational distribution is parametrized as diagonal Gaussian, the objective is defined as: where z t = f enc (o t) and σ 2 t = f enc std (o t). The latent vector z t is then used by an RL algorithm, such as SAC, instead of the unavailable true state s t. To infer temporal statistics, such as velocity and acceleration, it is common practice to stack three consecutive frames to form a single observation . We emphasize that in contrast to model-based methods , we do not predict future states and solely focus on learning representations from the current observation to stay model-free. In this section we explore in a systematic fashion how model-free off-policy RL can be made to train directly from pixel observations. We start by noting a dramatic performance drop when SAC is trained on pixels instead of proprioceptive state (Section 4.2) in the off-policy regime. This motivates us to explore different ways of employing auxiliary supervision to speed up representation learning. While a wide range of auxiliary objectives could be added to aid effective representation learning, for simplicity we focus our attention on autoencoders. We follow; and in Section 4.3 try an iterative unsupervised pretraining of an autoencoder that reconstructs pixels and is parameterized by β-VAE as per; Higgins et al. (2017a). Exploring the training procedure used in previous work shows it to be sub-optimal and points towards the need for end-to-end training of the β-VAE with the policy network. Our investigation in Section 4.4 renders this approach useless due to severe instability in training, especially with larger β values. We resolve this by using deterministic forms of the variational autoencoder and a careful learning procedure. This leads to our algorithm, which is described and evaluated in Section 5. We briefly state our setup here, for more details refer to Appendix B. Throughout the paper we evaluate on 6 image-based challenging continuous control tasks from depicted in Figure 1. For a concise presentation, in some places of the main paper we choose to plot for reacher easy, ball in cup catch, and walker walk only, while full are available in the Appendix. An episode for each task in maximum total reward of 1000 and lasts for exactly 1000 steps. Image observations are represented as 3 × 84 × 84 RGB renderings, where each pixel is scaled down to range. To infer velocity and acceleration we stack 3 consecutive frames following standard practice from. We keep the hyper parameters fixed across all tasks, except for action repeat, which we set only when learning from pixels according to for a fair comparison to the baselines. If action repeat is used, the number of training observations is only a fraction of the environment steps (e.g. a 1000 steps episode at action repeat 4 will only in 250 training observations). The exact action repeat settings can be found in Appendix B.3. We evaluate an agent after every 10000 training observation, by computing an average total reward across 10 evaluation episodes. For reliable comparison we run 10 random seeds for each configuration and compute mean and standard deviation of the evaluation reward. We start with an experiment comparing a model-free and off-policy algorithm SAC on pixels, with two state-of-the-art model-based algorithms, PlaNet and SLAC , and an upper bound of SAC on proprioceptive state (Table 1). We see a large gap between the capability of SAC on pixels (SAC:pixel), versus PlaNet and SLAC, which make use of many auxiliary tasks to learn a better representation, and can achieve performance close to the upper bound of SAC on proprioceptive state (SAC:state). From now, SAC:pixel will be our lower bound on performance as we gradually introduce different auxiliary reconstruction losses in order to close the performance gap. a) in the iterative re-training setup, we choose to employ a β-VAE likewise. We then proceed to first learn a representation space by pretraining the f enc, f enc std, and f dec networks of the β-VAE according to the loss J(VAE) Equation on data collected from a random policy. We then learn a control policy on top of the frozen latent representations z t = f enc (o t). We tune β for best performance, and find large β to be worse, and that very small β ∈ [10 −8, 10 −6] performed best. In Figure 2 we vary the frequency N at which the representation space is updated, from N = ∞, where the representation is never updated after an initial pretraining period with randomly collected data, to N = 1 where the representation is updated after every policy update. There is a positive correlation between this frequency and the final policy performance. We emphasize that the gradients are never shared between the β-VAE for learning the representation space, and the actor-critic learning the policy. These suggest that if we can combine the representation pretraining via a β-VAE together with the policy learning in a stable end-to-end procedure, we would expect better performance. However, we note that prior work (; a) with a regularized autoencoder to achieve stable end-to-end training from images in the off-policy regime. The stability comes from switching to a deterministic encoder that is carefully updated with gradients from the reconstruction J(AE) (Equation) and soft Q-learning J(Q) (Equation) objectives. Our findings and the from motivate us to allow gradient propagation to the encoder of the β-VAE from the actor-critic, which in our case is SAC. We enable end-to-end learning by allowing the encoder to not only update with gradients from the J(VAE) loss (Equation, as done in Section 4.3, but also with gradients coming from the J(Q) and J(π) (Equations and) losses specified in Section 3. Results in Figure 3 show that the end-to-end policy learning together with the β-VAE in unstable in the off-policy setting and prone to divergent behaviours that hurt performance. Our supports the findings from; Higgins et al. (2017a), which alleviate the problem by receding to the iterative re-training procedure. We next attempt stabilizing end-to-end training and introduce our method. We now seek to design a stable training procedure that can update the pixel autoencoder simultaneously with policy learning. We build on top of SAC , a model-free and off-policy actor-critic algorithm. Based on our findings from Section 4, we propose a new, simple algorithm, SAC+AE, that enables end-to-end training. We notice that electing to learn deterministic latent representations, rather than stochastic as in the β-VAE case, has a stabilizing effect on the end-to-end learning in the off-policy regime. We thus use a deterministic autoencoder in a form of the regularized autoencoder (RAE) , that has many structural similarities with β-VAE. We also found it is important to update the convolutional weights in the target critic network faster, than the rest of the parameters. This allows faster learning while preserving the stability of the off-policy actor-critic. Finally, we share the encoder's convolutional weights between the actor and critic networks, but prevent the actor from updating them. Our algorithm is presented in Figure 4 for visual guidance. We now show that our simple method, SAC+AE, achieves stable end-to-end training of an off-policy algorithm from images with an auxiliary reconstruction loss. We test our method on 6 challenging image-based continuous control tasks (see Figure 1) from DMC . The RAE consists of a convolutional and deconvolutional trunk of 4 layers of 32 filters each, with 3 × 3 kernel size. The actor and critic networks are 3 layer MLPs with ReLU activations and hidden size of 1024. We update the RAE and actor-critic network at each environment step with a batch of experience sampled from a replay buffer. A comprehensive overview of other hyper paremeters is Appendix B. We perform comparisons against several state-of-the-art model-free and model-based RL algorithms for learning from pixels. In particular: D4PG , an off-policy actor-critic algorithm, PlaNet , a model-based method that learns a dynamics model with deterministic and stochastic latent variables and employs cross-entropy planning for control, and SLAC , which combines a purely stochastic latent model together with an modelfree soft actor-critic. In addition, we compare against SAC that learns from low-dimensional pro- Figure 5: The main of our work. Our method demonstrates significantly improved performance over the baseline SAC:pixel. Moreover, it matches the state-of-the-art performance of model-based algorithms, such as PlaNet and SLAC , as well as a model-free algorithm D4PG , that also learns from raw images. Our algorithm exhibits stable learning across ten random seeds and is extremely easy to implement. prioceptive state, as an upper bound on performance. In Figure 5 we show that SAC+AE:pixel is able to match the state-of-the-art model-based methods such as PlaNet and SLAC, and significantly improve performance over the baseline SAC:pixel. Note that we use 10 random seeds, as recommended in whereas the PlaNet and SLAC numbers shown are only over 4 and 2 seeds, respectively, as per the original publications. To shed more light on some properties of the latent representation space learned by our algorithm we conduct several ablation studies. In particular, we want to answer the following questions: (i) is our method able to extract a sufficient amount of information from raw images to recover corresponding proprioceptive states readily? (ii) can our learned latent representation generalize to unseen tasks with similar image observations, but different reward objective, without reconstruction signal? Below, we answer these questions. Given how significantly our method outperforms a variant that does not have access to the image reconstruction signal, we hypothesize that the learned representation space encodes a sufficient amount of information about the internal state of the environment from raw images. Moreover, this information can be easily extracted from the latent state. To test this conjecture, we train SAC+AE:pixel and SAC:pixel until convergence on cheetah run, then fix their encoders. We then train two Encoder pretrained with our method (SAC+AE:pixel) on walker walk is able to generalize to unseen walker stand and walker run tasks. All three tasks share similar image observations, but have quite different reward structure. SAC with a pretrained on walker walk encoder achieves impressive final performance, while the baseline struggles to solve the tasks. identical linear projections to map the encoders' latent embedding of image observations into the corresponding proprioceptive states. Finally, we compare ground truth proprioceptive states against their reconstructions on a sample episode. Results in Figure 6 confirm our hypothesis that the encoder grounded on pixel observations is powerful enough to almost perfectly restore the internals of the task, whereas SAC without the reconstruction loss cannot. Full in Appendix F. To verify whether the latent representation space learned by our method is able to generalize to different tasks without additional fine-tuning with the reconstruction signal, we take three tasks walker stand, walker walk, and walker run from DMC, which share similar observational appearance, but have different reward structure. We train an agent using our method (SAC+AE:pixel) on walker walk task until convergence and extract its encoder. Consequently, we train two SAC agents without reconstruction loss on walker stand and walker run tasks from pixels. The encoder of the first agent is initialized with weights from the pretrained walker walk encoder, while the encoder of the second agent is not. Neither of the agents use the reconstruction signal, and only backpropogate gradients from the critic to the encoder (see Figure 4). Results in Figure 7 suggest that our method learns latent representations that can readily generalize to unseen tasks and help a SAC agent achieve strong performance and solve the tasks. We have presented the first end-to-end, off-policy, model-free RL algorithm for pixel observations with only reconstruction loss as an auxiliary task. It is competitive with state-of-the-art model-based methods, but much simpler, robust, and without requiring learning a dynamics model. We show through ablations the superiority of end-to-end learning over previous methods that use a two-step training procedure with separated gradients, the necessity of a pixel reconstruction loss over reconstruction to lower-dimensional "correct" representations, and demonstrations of the representation power and generalization ability of our learned representation. We find that deterministic models outperform β-VAEs (a), likely due to the other introduced instabilities, such as bootstrapping, off-policy data, and end-to-end training with auxiliary losses. We hypothesize that deterministic models that perform better even in stochastic environments should be chosen over stochastic ones with the potential to learn probability distributions, and argue that determinism has the benefit of added interpretability, through handling of simpler distributions. In the Appendix we provide across all experiments on the full suite of 6 tasks chosen from DMC (Appendix A), and the full set of hyperparameters used in Appendix B. There are also additional experiments autoencoder capacity (Appendix E), a look at optimality of the learned latent representation (Appendix H), importance of action repeat (Appendix I), and a set of benchmarks on learning from proprioceptive observation (Appendix J). Finally, we opensource our codebase for the community to spur future research in image-based RL. We evaluate the algorithms in the paper on the DeepMind control suite (DMC) -a collection of continuous control tasks that offers an excellent testbed for reinforcement learning agents. The software emphasizes the importance of having a standardised set of benchmarks with a unified reward structure in order to measure made progress reliably. Specifically, we consider six domains (see Figure 8) that in twelve different control tasks. Each task (Table 2) poses a particular set of challenges to a learning algorithm. The ball in cup catch task only provides the agent with a sparse reward when the ball is caught; the cheetah run task offers high dimensional internal state and action spaces; the reacher hard task requires the agent to explore the environment. We refer the reader to the original paper to find more information about the benchmarks. We employ double Q-learning (van) for the critic, where each Q-function is parametrized as a 3-layer MLP with ReLU activations after each layer except of the last. The actor is also a 3-layer MLP with ReLUs that outputs mean and covariance for the diagonal Gaussian that represents the policy. The hidden dimension is set to 1024 for both the critic and actor. We employ an almost identical encoder architecture as in , with two minor differences. Firstly, we add two more convolutional layers to the convnet trunk. Secondly, we use ReLU activations after each conv layer, instead of ELU. We employ kernels of size 3 × 3 with 32 channels for all the conv layers and set stride to 1 everywhere, except of the first conv layer, which has stride 2. We then take the output of the convnet and feed it into a single fully-connected layer normalized by LayerNorm . Finally, we add tanh nonlinearity to the 50 dimensional output of the fully-connected layer. The actor and critic networks both have separate encoders, although we share the weights of the conv layers between them. Furthermore, only the critic optimizer is allowed to update these weights (e.g. we truncate the gradients from the actor before they propagate to the shared conv layers). The decoder consists of one fully-connected layer that is then followed by four deconv layers. We use ReLU activations after each layer, except the final deconv layer that produces pixels representation. Each deconv layer has kernels of size 3 × 3 with 32 channels and stride 1, except of the last layer, where stride is 2. We then combine the critic's encoder together with the decoder specified above into an autoencoder. Note, because we share conv weights between the critic's and actor's encoders, the conv layers of the actor's encoder will be also affected by reconstruction signal from the autoencoder. We first collect 1000 seed observations using a random policy. We then collect training observations by sampling actions from the current policy. We perform one training update every time we receive a new observation. In cases where we use action repeat, the number of training observations is only a fraction of the environment steps (e.g. a 1000 steps episode at action repeat 4 will only into 250 training observations). The action repeat used for each environment is specified in Table 3, following those used by PlaNet and SLAC. We evaluate our agent after every 10000 environment steps by computing an average episode return over 10 evaluation episodes. Instead of sampling from the Gaussian policy we take its mean during evaluation. We preserve this setup throughout all the experiments in the paper. Action repeat cartpole swingup 8 reacher easy 4 cheetah run 4 finger spin 2 ball in cup catch 4 walker walk 2 Table 3: Action repeat parameter used per task, following PlaNet and SLAC. We initialize the weight matrix of fully-connected layers with the orthogonal initialization and set the bias to be zero. For convolutional and deconvolutional layers we use deltaorthogonal initialization . We regularize the autoencoder network using the scheme proposed in. In particular, we extend the standard reconstruction loss for a deterministic autoencoder with a L 2 penalty on the learned representation z and add weight decay on the decoder parameters θ dec:. We set λ z = 10 −6 and λ θ = 10 −7. We construct an observational input as an 3-stack of consecutive frames , where each frame is a RGB rendering of size 3 × 84 × 84 from the 0th camera. We then divide each pixel by 255 to scale it down to range. For reconstruction targets we instead preprocess images by reducing bit depth to 5 bits as in. We also provide a comprehensive overview of all the remaining hyper parameters in Temperature Adam's β 1 0.5 Init temperature 0.1 Table 4: A complete overview of used hyper parameters. Iterative pretraining suggested in; allows for faster representation learning, which consequently boosts the final performance, yet it is not sufficient enough to fully close the gap and additional modifications, such as end-to-end training, are needed. Figure 9 provides additional for the experiment described in Section 4.3.: An unsuccessful attempt to propagate gradients from the actor-critic down to the encoder of the β-VAE to enable end-to-end off-policy training. The learning process of SAC+VAE:pixel exhibits instability together with the subpar performance comparing to the baseline SAC+VAE:pixel (iter, 1), which does not share gradients with the actor-critic. We also investigate various autoencoder capacities for the different tasks. Specifically, we measure the impact of changing the capacity of the convolutional trunk of the encoder and corresponding deconvolutional trunk of the decoder. Here, we maintain the shared weights across convolutional layers between the actor and critic, but modify the number of convolutional layers and number of filters per layer in Figure 11 across several environments. We find that SAC+AE is robust to various autoencoder capacities, and all architectures tried were capable of extracting the relevant features from pixel space necessary to learn a good policy. We use the same training and evaluation setup as detailed in Appendix B.3. Learning from low-dimensional proprioceptive observations achieves better final performance with greater sample efficiency (see Figure 5 for comparison to pixels and Appendix J for proprioceptive baselines), therefore our intuition is to directly use these compact observations as the reconstruction targets to generate an auxiliary signal. Although, this is an unrealistic setup, given that we do not have access to proprioceptive states in practice, we use it as a tool to understand if such supervision is beneficial for representation learning and therefore can achieve good performance. We augment the observational encoder f enc, that maps an image o t into a latent vector z t, with a state decoder f state dec, that restores the corresponding state s t from the latent vector z t. This leads to an auxililary objective E ot,st∼D 1 2 ||f state dec (z t)−s t || 2 2, where z t = f enc (o t). We parametrize the state decoder f state dec as a 3-layer MLP with 1024 hidden size and ReLU activations, and train it end-to-end with the actor-critic network. Such auxiliary supervision helps less than expected, and surprisingly hurts performance in ball in cup catch, as seen in Figure 13. Our intuition is that such lowdimensional supervision is not able to provide the rich reconstruction error needed to fit the highcapacity convolutional encoder f enc. We thus seek for a denser auxiliary signal and try learning latent representation spaces with pixel reconstructions.: An auxiliary signal is provided by reconstructing a low-dimensional state from the corresponding image observation. Perhaps surprisingly, such synthetic supervision doesn't guarantee sufficient signal to fit the high-capacity encoder, which we infer from the suboptimal performance of SAC:pixel (state supervision) compared to SAC:pixel in ball in cup catch. We define the optimality of the learned latent representation as the ability of our model to extract and preserve all relevant information from the pixel observations sufficient to learn a good policy. For example, the proprioceptive state representation is clearly better than the pixel representation because we can learn a better policy. However, the differences in performance of SAC:state and SAC+AE:pixel can be attributed not only to the different observation spaces, but also the difference in data collected in the replay buffer. To decouple these attributes and determine how much information loss there is in moving from proprioceptive state to pixel images, we measure final task reward of policies learned from the same fixed replay buffer, where one is trained on proprioceptive states and the other trained on pixel observations. We first train a SAC+AE policy until convergence and save the replay buffer that we collected during training. Importantly, in the replay buffer we store both the pixel observations and the corresponding proprioceptive states. Note that for two policies trained on the fixed replay buffer, we are operating in an off-policy regime, and thus it is possible we won't be able to train a policy that performs as well. Figure 14: Training curves for the policy used to collect the buffer (SAC+AE:pixel (collector)), and the two policies learned on that buffer using proprioceptive (SAC:state (fixed buffer)) and pixel observations (SAC+AE:pixel (fixed buffer)). We see that our method actually outperforms proprioceptive observations in this setting. In Figure 14 we find, surprisingly, that our learned latent representation outperforms proprioceptive state on a fixed buffer. This could be because the data collected in the buffer is by a policy also learned from pixel observations, and is different enough from the policy that would be learned from proprioceptive states that SAC:state underperforms in this setting. We found that repeating nominal actions several times has a significant effect on learning dynamics and final reward. Prior works treat action repeat as a hyper parameter to the learning algorithm, rather than a property of the target environment. Effectively, action repeat decreases the control horizon of the task and makes the control dynamics more stable. Yet, action repeat can also introduce a harmful bias, that prevents the agent from learning an optimal policy due to the injected lag. This tasks a practitioner with a problem of finding an optimal value for the action repeat hyper parameter that stabilizes training without limiting control elasticity too much. To get more insights, we perform an ablation study, where we sweep over several choices for action repeat on multiple control tasks and compare acquired against PlaNet with the original action repeat setting, which was also tuned per environment. We use the same setup as detailed in Appendix B.3. Specifically, we average performance over 10 random seeds, and reduce the number of training observations inverse proportionally to the action repeat value. The are shown in Figure 15. We observe that PlaNet's choice of action repeat is not always optimal for our algorithm. For example, we can significantly improve performance of our agent on the ball in cup catch task if instead of taking the same nominal action four times, as PlaNet suggests, we take it once or twice. The same is true on a few other environments. Figure 15: We study the importance of the action repeat hyper parameter on final performance. We evaluate three different settings, where the agent applies a sampled action once (SAC+AE:pixel), twice (SAC+AE:pixel), or four times (SAC+AE:pixel). As a reference, we also plot the PlaNet with the original action repeat setting. Action repeat has a significant effect on learning. Moreover, we note that the PlaNet's choice of hyper parameters is not always optimal for our method (e.g. it is better to apply an action only once on walker walk, than taking it twice). In addition to the when an agent learns from pixels, we also provide a comprehensive comparison of several state-of-the-art continuous control algorithms that directly learn from proprioceptive states. Specifically, we consider four agents that implement SAC , TD3 , DDPG , and D4PG . We leverage open-source implementations of TD3 and DDPG from https://github.com/sfujim/TD3, and use the reported set of optimal hyper parameters, except of the batch size, which we increase to 512, as we find it improves performance of both the algorithms. Due to lack of a publicly accessible implementation of D4PG, we take the final performance after 10 8 environments steps as reported in. We use our own implementation of SAC together with the hyper parameters listed in Appendix B, again we increase the batch size to 512. Importantly, we keep the same set of hyper parameters across all tasks to avoid overfitting individual tasks. For this evaluation we do not repeat actions and perform one training update per every environment step. We evaluate a policy every 10000 steps (or every 10 episodes as one episode consists of 1000 steps) by running 10 evaluation episodes and averaging corresponding returns. To assess the stability properties of each algorithm and produce reliable baselines we compute mean and std of evaluation performance over 10 random seeds. We test on twelve challenging continuous control tasks from DMC Figure 16: We benchmark SAC, TD3, DDPG, and D4PG when learning from proprioceptive states on multiple tasks of various difficulty from DMC. We run 10 random seeds for each algorithm and evaluate on 10 trajectories (except for D4PG, as its implementation is not publicly available). We then report mean and standard deviation. For D4PG we take the performance after 10 8 environment steps reported in. We observe that SAC demonstrates superior performance and sample efficiency over the other methods on all of the tasks.

Represent the TLDR for finding the corresponding scientific article.

We design a simple and efficient model-free off-policy method for image-based reinforcement learning that matches the state-of-the-art model-based methods in sample efficiency

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.


Question: Represent the scientific article to find a short TLDR!

Question: Large deep neural networks require huge memory to run and their running speed is sometimes too slow for real applications. Therefore network size reduction with keeping accuracy is crucial for practical applications. We present a novel neural network operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible. Chopout is easy to implement and integrate into most type of existing neural networks. Furthermore it enables to reduce size of networks and latent representations even after training just by truncating layers. We show its effectiveness through several experiments. where where grad is a gradient and m is the number drawn in the forward pass. DISPLAYFORM0 At test time, chopout is defined to behave as a identity function, that is, just pass through the input 45 vector without any modification. This definition of chopout in prediction mode is contrastive to that 46 of dropout, which, in prediction time, dropout scale inputs to make it consistent with training time. Training a fully-connected neural network with applying chopout can be interpreted as simultaneous 48 training of randomly sampled sub-networks which are obtained by cuttinng out former parts of the 49 original fully-connected neural network with sharing parameters. In higher dimensional cases, chopout can be easily extended as a random truncation of channels 51 instead of dimensions. For example, when applied to a tensor x ∈ R c×h×w, the forward-propagation 52 of chopout is defined as DISPLAYFORM0 where P ({0, 1, · · ·, c}) is an arbitrary distribution. Back-propagation is defined in the same way. Throughout experiments, we use uniform distributions over {1, · · ·, d} for P m ({0, 1, · · ·, d}). We train autoencoders on MNIST (LeCun et al., Table 1, FIG4 ). We see that by applying 58 chopout on the hidden layer of the autoencoder, the reconstruction is kept well even after the hidden 59 layer is truncated. We apply chopout for embeddings trained through skip-gram models (Mikolov et al. [2013a,b] ). We 62 use text8 corpus 1. We set the window size to 5 and ignore infrequent words which appear less than 63 20 times in the corpus. The TAB2 shows the consistency of embeddings. ran ran news news good The distribution P m ({0, 1, · · ·, d}) should be explored. If we put chopouts in every layer of a 79 neural network, then, in training, there could be a layer where drawn m ∼ P m ({0, 1, · · ·, d}) is very 80 small and it could be a bottleneck of the prediction accuracy. Chopout can be used for network pruning.

We present a novel simple operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.