Question: Represent the scientific article to find a short TLDR

Question: We propose a new model for making generalizable and diverse retrosynthetic reaction predictions. Given a target compound, the task is to predict the likely chemical reactants to produce the target. This generative task can be framed as a sequence-to-sequence problem by using the SMILES representations of the molecules. Building on top of the popular Transformer architecture, we propose two novel pre-training methods that construct relevant auxiliary tasks (plausible reactions) for our problem. Furthermore, we incorporate a discrete latent variable model into the architecture to encourage the model to produce a diverse set of alternative predictions. On the 50k subset of reaction examples from the United States patent literature (USPTO-50k) benchmark dataset, our model greatly improves performance over the baseline, while also generating predictions that are more diverse. This paper proposes a novel approach for one-step retrosynthesis. This task is crucial for material and drug manufacturing and aims to predict which reactants are needed to generate a given target molecule as the main product. For instance, Figure 1 demonstrates that the input molecule "[N-]=[N+]=NCc1ccc(SCCl)cc1", expressed here as a SMILES string , can be generated using reactants "CSc1ccc(CN= [N+] =[N-])cc1" and "ClCCl". For decades, this task has been solved using template-based approaches . Templates encode transformation rules as regular expressions operating on SMILES strings and are typically extracted directly from the available training reactions. The primary limitation of such templates is coverage, i.e., it is possible that none of the templates applies to a test molecule. In order to better generalize to newer or broader chemical spaces, recently developed template-free approaches cast the problem as a sequence-to-sequence prediction task. These approaches were first explored by using LSTM models; the current state-of-the-art performance on this task uses Transformer models . Out-of-the-box Transformers nevertheless do not effectively generalize to rare reactions. For instance, model accuracy drops by 25% on reactions with 10 or fewer representative instances in the Figure 1: An example prediction task: on the left is the input target SMILES, and on the right are the output reactants SMILES. The input is a single molecule, while the output is a set of molecules separated by a period ("."). training set. 1 Another key issue is diversity. Manufacturing processes involve a number of additional criteria -such as green chemistry (having low detrimental effects on the environment). It is therefore helpful to generate a diverse collection of alternative ways of synthesizing the given target molecule. However, predicted reactions are unlikely to encompass multiple reaction classes (see Figure 2) without additional guidance. This is because the training data only provides a single reactant set for each input target, even if this is not the only valid reaction to synthesize the target. Figure 2: For the input target compound shown on the left, three possible reactant predictions are shown on the right. Prediction 1 suggestions a heterocycle formation reaction, while Predictions 2 and 3 both suggest substitution reactions. The only difference between the latter two is the halide functional group (Cl vs Br) highlighted in red. They share similar chemical properties and thus provide no additional insights for chemists. We extend molecular Transformers to address both of these challenges. First, we propose a novel pre-training approach to drive molecular representations to better retain alternative reaction possibilities. Our approach is reminiscent of successful pre-training schemes in natural language processing (NLP) applications . However, rather than using conventional token masking methods, we adopt chemically-relevant auxiliary tasks. Each training instance presents a single way to decompose a target molecule into its reactants. Here, we add alternative proxy decompositions for each target molecule by either 1) randomly removing bond types that can possibly break during reactions, or 2) transforming the target based on templates. While neither of these two auxiliary tasks are guaranteed to construct valid chemical reactions, they are closely related to the task of interest. Indeed, representations trained in this manner provide useful initializations for the actual retrosynthesis problem. To improve the diversity of predicted reactions, we incorporate latent variables into the generation process. Specifically, we merge the Transformer architecture with a discrete mixture over reactions. The role of the latent variable is to encode distinct modes that can be related to underlying reaction classes. Even though the training data only presents one reaction for each target molecule, our model learns to associate each reaction with a latent class, and in the process covers multiple reaction classes across the training set. At test time, a diverse collection of reactions is then obtained by collecting together predictions ing from conditioning on each latent class. Analogous mixture models have shown promise in generating diverse predictions in natural language translation tasks . We demonstrate similar gains in the chemical context. We evaluate our model on the benchmark USPTO-50k dataset, and compare it against state-ofthe-art template-free baselines using the Transformer model. We focus our evaluation on top-10 accuracy, because there are many equally valuable reaction transformations for each input target, though only one is presented in the data. Compared to the baseline, we achieve better performance overall, with over 13% increase in top-10 accuracy for our best model. When we create a split of the data based on different reaction templates (a task that any template-based model would fail on), we similarly observe a performance increase for our model. Additionally, we demonstrate that our model outputs exhibit significant diversity through both quantitative and human evaluations. Template-based Models Traditional methods for retrosynthetic reaction prediction use templatebased models. Templates, or rules, denote the exact atom and bond changes for a chemical reaction. applies these templates for a given target compound based on similar reactions in the dataset. Going one step further, learns the associations between molecules and templates through a neural network. uses a hierarchical network to first predict the reaction group and then the correct template for that group. However, to have the flexibility to generalize beyond extracted rules, we explore template-free generative models. Molecule Generation There are two different approaches to generative tasks for molecules, demonstrated through graph and SMILES representations. The graph-generation problem has been explored in as a node-by-node generation algorithm, but this model does not guarantee the validity of the output chemical graph. Jin et al. (2018a; b) improves upon this method using a junction-tree encoder-decoder that forces the outputs to be constrained in the valid chemical space; however, these models require complex, structured decoders. We focus on the generative task of SMILES string representations of the molecules, which has been explored in and Gómez-. Pre-training Pre-training methods have been shown to vastly improve the performance of Transformer models in NLP tasks without additional data supervision. use a masked language modeling objective to help their model learn effective representations for downstream tasks. Similar pre-training methods on molecules have been explored by , where they mask out atoms in molecular graphs. Meanwhile, our work does not use a masked objective, but instead creates pre-training tasks that are relevant to the retrosynthesis prediction problem. Given an input target molecule, the task of retrosynthetic reaction prediction is to output likely reactants that can form the target product. Formally, we express a molecule as a text string via its SMILES representation, and cast our task into a sequence-to-sequence (seq2seq) prediction problem (example shown in Figure 1). For this task, the input target is always a single molecule, while the output predictions are usually a set of more than one molecule concatenated by separators ".". To provide more intuition for this generative task, we describe some properties of SMILES strings. Each SMILES string is 1-D encoding of a 2-D molecular graph. If the predicted SMILES does not adhere to the SMILES grammar, then a valid molecular graph cannot be reconstructed. Moreover, each molecule has many equivalent SMILES representations, as a single instance of its SMILES is just a graph traversal starting at some arbitrary node. Therefore, two very different SMILES string can encode the same molecule (see Appendix B), and the model needs to be robust to the given input. One method, proposed by , augments the input data with different SMILES strings of the same input target molecule. For our model architecture, we apply a Transformer model for the seq2seq task, which has an encoder-decoder structure . The encoder maps an input sequence of tokens (from the SMILES string) to a sequence of continuous representations, which are then fed to the decoder to generate an output sequence of tokens one element at a time, auto-regressively. Once the model is trained, a beam search procedure is used at inference time to find likely output sequences. The main building block of the Transformer architecture lies in its global self-attention layers, which are well-suited for predictions of the latent graph structure of SMILES strings. For example, two tokens that are far apart in the SMILES string could be close together topologically in the corresponding molecular graph. The global connectivity of the Transformer model allows it to better leverage this information. Additionally, since SMILES follow a rigid grammar requiring long rangedependencies, these dependencies can be more easily learned through global attention layers (see Appendix B). Despite the flexible architecture of Transformer models, we recognize that there are ways to improve model generalization. Additionally, there is no inductive bias for proposing diverse outputs. We propose two techniques to enhance the base molecular Transformer model, which we describe now. In the data, each input target molecule is associated with a single reaction transformation -though there are many equally good chemical reactions. Therefore, for each input target, we construct several new prediction examples that are chemically meaningful, and pre-train the model on these auxiliary examples. We do so without requiring additionally data, or data supervision. The two variants of our method are described in detail below, with examples shown in Figure 3. with two automatically generated pre-training targets formed by breaking the bond highlighted in red. Examples and are generated from the random and template-based methods respectively. The only difference is that the template-based pre-training example adds an additional function group to the molecule (blue). Random pre-training For each input target molecule, we generate new examples by selecting a random bond to break. The types of bonds that we consider are acyclic single bonds, because these are the bonds most commonly broken in chemical reactions. As we break an acyclic bond, the input molecule is necessarily broken up into two output molecules, each being a subgraph of the input molecule. Although the examples generated by this method do not cover the entire space of chemical reactions (for instance some reactions do not break any bonds at all), these examples are easy to generate and cover a diverse range of transformations. Template-based pre-training Instead of randomly breaking bonds, we can also use the templates extracted from the training data to create reaction examples. An example of a template is shown in Figure 4: each template matches a specific pattern in the input molecule, and transforms that pattern according to the template specifications. When the matched pattern is a single acyclic bond, this method will generate similar outputs as the random pre-training method, except that templates usually add additional pieces (functional groups) to the output example. Figure 4: An example of a template, where the exact bond changes are described in red. The "C-N" bond (left) is broken and a "Br" atom is attached to the broken "C" atom (right). As shown in Figure 3, both examples are derived from the same bond broken in the input target molecule, but for the template-based example, an additional functional group was added, matching a more realistic reaction context. On average, for a random input molecule, there are 10 different possible examples that can be extracted from the random pre-training method, while there are over 200 different possible examples that can be extracted using the template-based pre-training method. However, many of these 200 examples represent similar chemical transformations, only differing in the type of functional group added. More broadly speaking, we can say that the template pre-training method generates more chemically valid reactions compared to the random pre-training method. However, the advantage of the random pre-training method is that it can break bonds that are not represented within the templates, thereby perhaps conferring a higher ability to generalize. As routine, the model is pre-trained on these automatically constructed auxiliary tasks, and then used as initialization to be fine-tuned on the actual retrosynthesis data. Next, we tackle the problem of generating diverse predictions. As mentioned earlier, the retrosynthesis problem is a one-to-many mapping since a target molecule can be formed from different types of reactions. We would like the model to produce a diverse set of predictions so that chemists can choose the most feasible and economical one in practice. However, hypotheses generated by a vanilla seq2seq model with beam search typically exemplifies low diversity with only minor differences in the suffix, see Figure 8 . To address this, we use a mixture seq2seq model that has shown sucess in generating diverse machine translations to generate diverse retrosynthesis reaction predictions . Specifically, given a target SMILES string x and reactants SMILES string y, a mixture model introduces a multinomial latent variable z ∈ {1, · · ·, K} to capture different reaction types, and decomposes the marginal likelihood as: Here, the prior p(z|x; θ) and likelihood p(y|z, x; θ) parameterized by θ are functions to be learned. We use a uniform prior p(z|x; θ) = 1/K, which is easy to implement and works well in practice . For p(y|z, x; θ), we share the encoder-decoder network among mixture components, and feed the embedding of z as an input to the decoder so that y is conditioned on it. The increase in the parameters of our model is negligible over the baseline model. We train the mixture model with the online hard-EM algorithm. Taking a mini-batch of training examples {(, we enumerate all K values of z and compute their loss, − log p(y (i) |z, x (i); θ). Then, for each (x (i), y (i) ), we select the value of z that yields the minimum loss:, and back-propagate through it, so only one component receives gradients per example. An important detail for successfully training a mixture model is that dropout is turned off in the forward passes for latent variable selection, and turned back on at back-propagation time for gradient regularization. Otherwise even a small amount of dropout noise will corrupt the optimal value of z, making the selection random and the different latent components will fail to diversify . The hard selection of the latent variable forces different components to specialize on different subsets of the data. As we shall later see in the experimental , our mixture model can learn to represent different reaction types in the training data and show improved diversity over the baseline. The benchmark dataset we use is a subset of the open source patent database of chemical reactions . Specifically, we use the curated 50k subset (USPTO-50k) from , including the same data splits. Each example reaction in this dataset is labeled with one of ten reaction classes, which describes its transformation type, but we do not use this information in our experiments, similar to. Since we are only interested in the retrosynthesis prediction problem, the examples are processed to remove any reagent molecules (molecules that do not contribute atoms to the reaction). The reactions are tokenized in the same manner as in , with each token being a meaningful subunit of the molecule (i.e., an atom or bond). In addition, we create a separate split of the USPTO-50k data, in which the train and test sets are split by reaction templates. Specifically, we split the data so that no example in the test set can be solved correctly with any templates extracted from training data. We use the template extraction code from , which to the best of our knowledge, is the only publicly available template extraction code. Accuracy The evaluation of retrosynthesis is challenging, because each input target has many valid syntheses, but only one is given in the data. When the model output does not exactly match the single solution in the data, the model is not necessarily wrong, but simply giving a plausible alternative. Therefore, we focus on the top-10 accuracy for our evaluation, but present all from our experiments. We compute the accuracies by matching the canonical SMILES strings of molecule sets. For the mixture model, we output the top 10 predictions for each latent class, and then combine those based on likelihoods to get top 10 predictions overall. Diversity To measure diversity, we provide both quantitative and human evaluations. For the former, we train a model to predict the reaction class given the input target molecule and the predicted output. We use a typical message-passing graph convolution network to embed both the input and output molecules (using weight-sharing) and compute the reaction embedding as a difference of the two embeddings. This predictor is trained on the 10 reaction class labels in the USPTO-50k dataset, and achieves 99% accuracy on the test set, so we can be fairly confident in its ability to predict the reaction class in-domain. Our main baseline is the SMILES transformer (Base), adapted from. We run the same model as other recent works for this task , and we build on top of the Transformer implementation from OpenNMT . We run ablation experiments for pre-training and different mixture models to show the impact of each approach. Random pre-training is referred to as Pre-train (R), while template-based pre-training is referred to as Pre-train (T). For each example, we construct up to 10 new auxiliary examples, and pre-train the model on these examples. Additionally, following , we also augment the training data with variations of the input SMILES string, referred to as Aug. That is, for each training example, we add an extra example using a different input SMILES string, which is trained to predict the same output reactants. This helps the model learn representations robust to the permutation of the input SMILES string. In addition to our experiments, we include a templatebased approach from , and a template-free approach from that adds a syntax predictor on top of the transformer model. Accuracy The accuracy of our model is shown in Table 1. We observe that both pre-training tasks improve over the baseline, and more so when combined with data augmentation. This shows that our pre-training tasks help the model learn the chemical reaction representational space, and are useful for the retrosynthesis prediction problem. However, interestingly, there seem to be marginal differences between the two pre-training methods. We attribute this to the fact that both pre-training methods usually generate very similar sets of examples. Previously shown in Figure 3, one of the main differences of template-based pre-training is just that it adds additional functional groups. But since these generated examples are not always chemically valid, having this extra information may not prove to be very valuable. We do note, however, that constructing additional decompositions of the input targets does actually matter for the pre-training task. We had also experimented with pretraining methods that only used variations of the input SMILES strings as pre-training output targets (because each molecule has many different SMILES representations). However, these methods did not in the same performance gains, because these pre-training targets do not contain much useful information for the actual task. Our original motivation for using a mixture model was to improve diversity, but we observe that it also leads to an increase in performance. We try N = {1, 2, 5} for the number of discrete latent classes, and we see that more latent classes generally leads to higher accuracies. The top-1 accuracy does decrease slightly as the number of latent classes increases, but we observe much higher accuracies at top-10 (increase of 7-8%). Importantly, we note that our method of combining outputs from different latent classes is not perfect, as the likelihoods from different latent classes are not totally comparable. That is likely the cause of the decrease in top-1 accuracy; yet as we mentioned in Section 5.2, top-10 accuracies are significantly more meaningful for our problem. Table 2: Prediction accuracies when tested on our template split of the USPTO dataset, for which any template-based model would get 0% accuracy on the test set. We see that our template-free methods can still generalize to this test set. Next, we show our on a different split of the data, which is more challenging to generalize. Using the dataset split on templates described in Section 5.1, we explore the performance of our best mixture model with pre-training compared to the baseline with no pre-training. As mentioned earlier, template-free models confer advantages over template-based models, as template-based models lack the ability to generalize outside of extracted rules. For this dataset, any template-based model would necessarily achieve 0% accuracy based on construction. Diversity We now look at evaluations of diversity for our model. Using the reaction class model described in Section 5.2, we predict the reaction class for every output of our models. Then, we compute the average number of unique reaction classes, holding all other factors constant besides varying the number of latent classes ( shown in Table 3). The number of unique reaction classes is 3.32 for the mixture model compared to the 2.66 for the base model, suggesting that the mixture model predicts a more diverse cast of outputs. The diversity of the predictions can also be examined from an interpretability standpoint for the latent classes of the mixture model. Using the reaction class model, we take the 10 top predictions from each latent class, and count the number of occurrences for each reaction class. Normalizing across reaction classes, we can see from Figure 6 that each latent class learns to predict a different distribution of reaction classes. We also supplement our diversity with human evaluation. To make the problem tractable for a human chemist, we randomly select 100 different reactions from the test set and present the top 5 predicted outputs from both the base and mixture model, where the the task is to determine diversity based on number of different types of reactions. The human chemist is asked to choose which of the two output sets is more diverse, or neither if the two sets do not differ in diversity (see Appendix C). For this task, the human chemist chose the mixture model more than twice as often as the base model (43 times vs 21), see Table 3. Although not perfect, these exemplify that our model does generate more diverse outputs than the baseline. We explored the problem of making one-step retrosynthesis reaction predictions, dealing with the issues of generalizability and making diverse predictions. Through pre-training and use of mixture models, we show that our model beats state-of-the-art methods in terms of accuracy and generates more diverse predictions. Even on a challenging task, for which any template-based models would fail, our model still is able to generalize to the test set. To compute a subset of the data with only rare reactions, we extracted all the templates from the entire USPTO-50k dataset, and selected the templates that occurs at most 10 times. The reactions in the test set that have these templates constitute the rare reaction subset, which is around 400 examples. The for this rare reaction subset can be found in Table 4. From this table, we can see that the top-1 accuracy for the baseline model is only 18.6% which is roughly 25% drop from the 42% in Table 1. We also mention that our new models improve over this baseline, showing more generalizability. Table 4: Prediction accuracies on the rare reaction test subset. Each molecule has many different SMILES representation, because each different SMILES string is just a different graph traversal over the molecule (see Figure 7). Although there is often some canonical SMILES string which is consistent, it is still completely arbitrary. Additionally, because SMILES is a 1-D encoding of the 2-D molecular graph, two atoms that are close in the graph may be far apart in the SMILES string, shown in Figure 8. To correctly decode a SMILES string, the decoder has to be aware of long-range dependencies. For instance, numbers in the SMILES string indicate the start and end of a cycle. The decoder has to close all cycles that it starts, and at the right position, or else the output SMILES will be invalid. Figure 8: The carbon atom (red) and the oxygen atom (blue) are neighbors on the molecular graph. However, in the SMILES string, they are far apart.

Represent the TLDR for finding the corresponding scientific article

We propose a new model for making generalizable and diverse retrosynthetic reaction predictions.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Unsupervised learning of disentangled representations is an open problem in machine learning. The Disentanglement-PyTorch library is developed to facilitate research, implementation, and testing of new variational algorithms. In this modular library, neural architectures, dimensionality of the latent space, and the training algorithms are fully decoupled, allowing for independent and consistent experiments across variational methods. The library handles the training scheduling, logging, and visualizations of reconstructions and latent space traversals. It also evaluates the encodings based on various disentanglement metrics. The library, so far, includes implementations of the following unsupervised algorithms VAE, Beta-VAE, Factor-VAE, DIP-I-VAE, DIP-II-VAE, Info-VAE, and Beta-TCVAE, as well as conditional approaches such as CVAE and IFCVAE. The library is compatible with the Disentanglement Challenge of NeurIPS 2019, hosted on AICrowd and was used to compete in the first and second stages of the challenge, where it was ranked among the best few participants. There are two overlapping avenues in representation learning. One focuses on learning task-specific transformations often optimized towards specific domains and applications. The other approach learns the intrinsic factors of variation, in a disentangled and taskinvariant fashion. The unsupervised disentanglement of latent factors, where changes in a single factor of variation shifts the latent encoding in a single direction, is an open problem of representation learning . Disentangled representations are valuable in few-shot learning, reinforcement learning, transfer learning, as well as semisupervised learning (; ; . In this work, we developed a library based on the functionalities of the PyTorch framework, which facilitates research, implementation, and testing of new variational algorithms focusing on representation learning and disentanglement. The library branches from the Disentanglement Challenge of NeurIPS 2019, hosted on AICrowd (aicrowd.com), and was used to compete in the first and second stages of the challenge where it was highly ranked. The Disentanglement-PyTorch library is released under the GNU General Public License at https://github.com/amir-abdi/disentanglement-pytorch. Unsupervised Objectives Currently, the library includes implementations of the following unsupervised variational algorithms: VAE , β-VAE , β-TCVAE , Factor-VAE , Info-VAE , DIP-I-VAE, and DIP-II-VAE . Algorithms are implemented as plug-ins to the variational Bayesian formulation, and are specified by the loss terms flag. As a , if the loss terms of two learning algorithms (e.g., A and B) were found to be compatible, they can both be included in the objective function with the flag set as [--loss terms A B]. This enables researchers to mix and match loss terms which optimize towards correlated goals. The library supports conditional approaches such as CVAE , where extra known attributes (i.e, labels) are included in the encoding and decoding processes. It also supports IFCVAE, inspired by the IFcVAE-GAN , which enforces certain latent factors to encode known attributes using a set of positive (auxiliary) and negative (adversarial) discriminators in a supervised fashion. Thanks to the modular implementation of the library, any of the above-mentioned unsupervised loss terms can be used with conditional and information factoriation approaches to encourage disentanglement across attribute-invariant latents. Neural architectures and the dimensionality of the data and the latent spaces are configurable and decoupled from the training algorithm. Consequently, new architectures for the encoder and decoder networks, such as the auto-regressive models, and support for other data domains, can be independently investigated. We rely on Google's implementation of the disentanglement metrics to evaluate the quality of the learned representations. Thanks to the disentanglement-lib 1 library, the following metrics are currently supported: BetaVAE , FactorVAE , Mutual Information Gap (MIG) , Interventional Robustness Score (IRS) , Disentanglement Completeness and Informativeness (DCP) , and Separated Attribute Predictability (SAP) . Controlled Capacity Increase It is shown that gradually relaxing the information bottleneck during training improves the disentanglement without penalizing the reconstruction accuracy. Following the formulation of , the capacity, defined as the distance between the prior and the latent posterior distributions and denoted with the variable C, is gradually increased during training. To avoid convergence points with high reconstruction loss, training can be started with more emphasis on the reconstruction and gradually relaxing for the disentanglement term to become more relative. Dynamic Learning Rate Scheduling All forms of learning rate schedulers are supported. Researchers are encouraged to leverage the dynamic LR scheduling to gradually decrease the rate when the average objective function over the epoch stops its decremental trend. Logging and Visualization The library leverages the Weights & Biases (W&B) 2 tool to record and visualize the training process and experiments' . Besides the scalar values, we visualize the attribute and condition traversals, latent factor traversals, and input reconstructions, both as static images (logged via W&B) as well as animated GIFs. The β-TCVAE algorithm achieved the best disentanglement on the mpi3d real dataset in the second stage of the disentanglement challenge. Given the limited 8-hour training time for the challenge, the model was pre-trained on the mpi3d toy dataset . The model was trained with the Adam optimizer for 90k iterations on batches of size 64. The β value of the β-TCVAE objective function was set to 2. The learning rate was initialized at 0.001 and reduced on the plateau of the objective function with a factor of 0.95. The capacity parameter, C, was gradually increased from 0 to 25. The dimensionality of the z space was generously set to 20. The encoder consisted of 5 convolutional layers with strides of 2, kernel sizes of 3 × 3, and number of kernels gradually increasing from 32 to 256. The encoder ended with a dense linear layer which estimated the posterior latent distribution as a parametric Gaussian. The decoder network consisted of one convolutional followed with 6 deconvolutional (transposed convolutional) layers, with kernel sizes of 4, strides of 2, and the number of kernels gradually decreasing from 256 down to the number of channels of the image space. ReLU activations were used except for the last layers of the encoder and decoder networks. Model's performance on the unseen objects of mpi3d realistic and mpi3d real datasets are presented in Table 1. The configurations of the two experiments are the same and the model consistently performed better on the mpi3d real dataset. This was unexpected as the model was initialized on the mpi3d toy dataset. Disentanglement performance, on the available samples of the mpi3d realistic dataset, is visualized in Appendix A. Figure 1: Latent factor traversal of the trained β-TCVAE model on a random sample of the mpi3d realistic dataset. As demonstrated, the disentanglement is not complete and some features are encoded in the same latent factor. A latent space of size 20 was used, however, changes in the other 13 latent factors had no effect on the reconstruction; thus, these feature-invariant factors were not included for brevity.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Disentanglement-PyTorch is a library for variational representation learning

Represent the TLDR for finding the corresponding scientific article

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.


Question: Represent the scientific article to find a short TLDR

Question: Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL, which exploits an observation that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. The introduction of relative entropy regularization allows Trust-PCL to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL significantly improves the solution quality and sample efficiency of TRPO. The goal of model-free reinforcement learning (RL) is to optimize an agent's behavior policy through trial and error interaction with a black box environment. Value-based RL algorithms such as Q-learning BID36 and policy-based algorithms such as actor-critic BID15 have achieved well-known successes in environments with enumerable action spaces and predictable but possibly complex dynamics, e.g., as in Atari games BID19 BID34. However, when applied to environments with more sophisticated action spaces and dynamics (e.g., continuous control and robotics), success has been far more limited. In an attempt to improve the applicability of Q-learning to continuous control, BID32 and BID16 developed an off-policy algorithm DDPG, leading to promising on continuous control environments. That said, current off-policy methods including DDPG often improve data efficiency at the cost of optimization stability. The behaviour of DDPG is known to be highly dependent on hyperparameter selection and initialization BID18; even when using optimal hyperparameters, individual training runs can display highly varying outcomes. On the other hand, in an attempt to improve the stability and convergence speed of policy-based RL methods, BID13 developed a natural policy gradient algorithm based on , which subsequently led to the development of trust region policy optimization (TRPO) BID28. TRPO has shown strong empirical performance on difficult continuous control tasks often outperforming value-based methods like DDPG. However, a major drawback is that such methods are not able to exploit off-policy data and thus require a large amount of on-policy interaction with the environment, making them impractical for solving challenging real-world problems. Efforts at combining the stability of trust region policy-based methods with the sample efficiency of value-based methods have focused on using off-policy data to better train a value estimate, which can be used as a control variate for variance reduction BID8 b).In this paper, we investigate an alternative approach to improving the sample efficiency of trust region policy-based RL methods. We exploit the key fact that, under entropy regularization, the optimal policy and value function satisfy a set of pathwise consistency properties along any sampled path BID21, which allows both on and off-policy data to be incorporated in an actor-critic algorithm, PCL. The original PCL algorithm optimized an entropy regularized maximum reward objective and was evaluated on relatively simple tasks. Here we extend the ideas of PCL to achieve strong on standard, challenging continuous control benchmarks. The main observation is that by alternatively augmenting the maximum reward objective with a relative entropy regularizer, the optimal policy and values still satisfy a certain set of pathwise consistencies along any sampled trajectory. The ing objective is equivalent to maximizing expected reward subject to a penalty-based constraint on divergence from a reference (i.e., previous) policy. We exploit this observation to propose a new off-policy trust region algorithm, Trust-PCL, that is able to exploit off-policy data to train policy and value estimates. Moreover, we present a simple method for determining the coefficient on the relative entropy regularizer to remain agnostic to reward scale, hence ameliorating the task of hyperparameter tuning. We find that the incorporation of a relative entropy regularizer is crucial for good and stable performance. We evaluate Trust-PCL against TRPO, and observe that Trust-PCL is able to solve difficult continuous control tasks, while improving the performance of TRPO both in terms of the final reward achieved as well as sample-efficiency. Trust Region Methods. Gradient descent is the predominant optimization method for neural networks. A gradient descent step is equivalent to solving a trust region constrained optimization, DISPLAYFORM0 which yields the locally optimal update dθ = −η∇ (θ) such that η = √ / ∇ (θ); hence by considering a Euclidean ball, gradient descent assumes the parameters lie in a Euclidean space. However, in machine learning, particularly in the context of multi-layer neural network training, Euclidean geometry is not necessarily the best way to characterize proximity in parameter space. It is often more effective to define an appropriate Riemannian metric that respects the loss surface , which allows much steeper descent directions to be identified within a local neighborhood (e.g., ; BID17). Whenever the loss is defined in terms of a Bregman divergence between an (unknown) optimal parameter θ * and model parameter θ, i.e., (θ) ≡ D F (θ *, θ), it is natural to use the same divergence to form the trust region: DISPLAYFORM1 The natural gradient is a generalization of gradient descent where the Fisher information matrix F (θ) is used to define the local geometry of the parameter space around θ. If a parameter update is constrained by dθ DISPLAYFORM2 is obtained. This geometry is especially effective for optimizing the log-likelihood of a conditional probabilistic model, where the objective is in fact the KL divergence D KL (θ *, θ). The local optimization is, DISPLAYFORM3 Thus, natural gradient approximates the trust region by DISPLAYFORM4, which is accurate up to a second order Taylor approximation. Previous work BID13 BID4 BID24 BID28 has applied natural gradient to policy optimization, locally improving expected reward subject to variants of dθ T F (θ)dθ ≤. Recently, TRPO BID28 has achieved state-of-the-art in continuous control by adding several approximations to the natural gradient to make nonlinear policy optimization feasible. Another approach to trust region optimization is given by proximal gradient methods BID23. The class of proximal gradient methods most similar to our work are those that replace the hard constraint in with a penalty added to the objective. These techniques have recently become popular in RL BID35 BID11 BID31, although in terms of final reward performance on continuous control benchmarks, TRPO is still considered to be the state-of-the-art. BID22 make the observation that entropy regularized expected reward may be expressed as a reversed KL divergence D KL (θ, θ *), which suggests that an alternative to the constraint in should be used when such regularization is present: DISPLAYFORM5 Unfortunately, this update requires computing the Fisher matrix at the endpoint of the update. The use of F (θ) in previous work can be considered to be an approximation when entropy regularization is present, but it is not ideal, particularly if dθ is large. In this paper, by contrast, we demonstrate that the optimal dθ under the reverse KL constraint D KL (θ + dθ, θ) ≤ can indeed be characterized. Defining the constraint in this way appears to be more natural and effective than that of TRPO.Softmax Consistency. To comply with the information geometry over policy parameters, previous work has used the relative entropy (i.e., KL divergence) to regularize policy optimization; ing in a softmax relationship between the optimal policy and state values BID25 BID3 BID2 BID7 BID26 under single-step rollouts. Our work is unique in that we leverage consistencies over multi-step rollouts. The existence of multi-step softmax consistencies has been noted by prior work-first by BID21 in the presence of entropy regularization. The existence of the same consistencies with relative entropy has been noted by BID30. Our work presents multi-step consistency relations for a hybrid relative entropy plus entropy regularized expected reward objective, interpreting relative entropy regularization as a trust region constraint. This work is also distinct from prior work in that the coefficient of relative entropy can be automatically determined, which we have found to be especially crucial in cases where the reward distribution changes dramatically during training. Most previous work on softmax consistency (e.g., BID7 ; Azar et al. FORMULA0 ; BID21) have only been evaluated on relatively simple tasks, including grid-world and discrete algorithmic environments. BID26 conducted evaluations on simple variants of the CartPole and Pendulum continuous control tasks. More recently, BID10 showed that soft Qlearning (a single-step special case of PCL) can succeed on more challenging environments, such as a variant of the Swimmer task we consider below. By contrast, this paper presents a successful application of the softmax consistency concept to difficult and standard continuous-control benchmarks, ing in performance that is competitive with and in some cases beats the state-of-the-art. We model an agent's behavior by a policy distribution π(a | s) over a set of actions (possibly discrete or continuous). At iteration t, the agent encounters a state s t and performs an action a t sampled from π(a | s t). The environment then returns a scalar reward r t ∼ r(s t, a t) and transitions to the next state s t+1 ∼ ρ(s t, a t). When formulating expectations over actions, rewards, and state transitions we will often omit the sampling distributions, π, r, and ρ, respectively. Maximizing Expected Reward. The standard objective in RL is to maximize expected future discounted reward. We formulate this objective on a per-state basis recursively as DISPLAYFORM0 The overall, state-agnostic objective is the expected per-state objective when states are sampled from interactions with the environment: DISPLAYFORM1 Most policy-based algorithms, including REINFORCE BID37 and actorcritic BID15, aim to optimize O ER given a parameterized policy. Path Consistency Learning (PCL). Inspired by BID37, BID21 augment the objective O ER in with a discounted entropy regularizer to derive an objective, DISPLAYFORM2 where τ ≥ 0 is a user-specified temperature parameter that controls the degree of entropy regularization, and the discounted entropy H(s, π) is recursively defined as DISPLAYFORM3 Note that the objective O ENT (s, π) can then be re-expressed recursively as, BID21 show that the optimal policy π * for O ENT and V * (s) = O ENT (s, π *) mutually satisfy a softmax temporal consistency constraint along any sequence of states s 0,..., s d starting at s 0 and a corresponding sequence of actions a 0,..., a d−1: DISPLAYFORM4 DISPLAYFORM5 This observation led to the development of the PCL algorithm, which attempts to minimize squared error between the LHS and RHS of to simultaneously optimize parameterized π θ and V φ. Importantly, PCL is applicable to both on-policy and off-policy trajectories. Trust Region Policy Optimization (TRPO). As noted, standard policy-based algorithms for maximizing O ER can be unstable and require small learning rates for training. To alleviate this issue, BID28 proposed to perform an iterative trust region optimization to maximize O ER. At each step, a prior policyπ is used to sample a large batch of trajectories, then π is subsequently optimized to maximize O ER while remaining within a constraint defined by the average per-state KL-divergence withπ. That is, at each iteration TRPO solves the constrained optimization problem, DISPLAYFORM6 The prior policy is then replaced with the new policy π, and the process is repeated. To enable more stable training and better exploit the natural information geometry of the parameter space, we propose to augment the entropy regularized expected reward objective O ENT in with a discounted relative entropy trust region around a prior policyπ, DISPLAYFORM0 where the discounted relative entropy is recursively defined as DISPLAYFORM1 This objective attempts to maximize entropy regularized expected reward while maintaining natural proximity to the previous policy. Although previous work has separately proposed to use relative entropy and entropy regularization, we find that the two components serve different purposes, each of which is beneficial: entropy regularization helps improve exploration, while the relative entropy improves stability and allows for a faster learning rate. This combination is a key novelty. Using the method of Lagrange multipliers, we cast the constrained optimization problem in into maximization of the following objective, DISPLAYFORM2 Again, the environment-wide objective is the expected per-state objective when states are sampled from interactions with the environment, DISPLAYFORM3 A key technical observation is that the O RELENT objective has a similar decomposition structure to O ENT, and one can cast O RELENT as an entropy regularized expected reward objective with a set of transformed rewards, i.e., DISPLAYFORM0 where DISPLAYFORM1 is an expected reward objective on a transformed reward distribution functioñ r(s, a) = r(s, a) + λ logπ(a|s). Thus, in what follows, we derive a corresponding form of the multi-step path consistency in.Let π * denote the optimal policy, defined as π * = argmax π O RELENT (π). As in PCL BID21, this optimal policy may be expressed as DISPLAYFORM2 where V * are the softmax state values defined recursively as DISPLAYFORM3 We may re-arrange to yield DISPLAYFORM4 This is a single-step temporal consistency which may be extended to multiple steps by further expanding V * (s t+1) on the RHS using the same identity. Thus, in general we have the following softmax temporal consistency constraint along any sequence of states defined by a starting state s t and a sequence of actions a t,..., a t+d−1: DISPLAYFORM5 We propose to train a parameterized policy π θ and value estimate V φ to satisfy the multi-step consistencies in. Thus, we define a consistency error for a sequence of states, actions, and rewards s t:t+d ≡ (s t, a t, r t, . . ., s t+d−1, a t+d−1, r t+d−1, s t+d) sampled from the environment as DISPLAYFORM0 We aim to minimize the squared consistency error on every sub-trajectory of length d. That is, the loss for a given batch of episodes (or sub-episodes) S = {s DISPLAYFORM1 We perform gradient descent on θ and φ to minimize this loss. In practice, we have found that it is beneficial to learn the parameter φ at least as fast as θ, and accordingly, given a mini-batch of episodes we perform a single gradient update on θ and possibly multiple gradient updates on φ (see Appendix for details).In principle, the mini-batch S may be taken from either on-policy or off-policy trajectories. In our implementation, we utilized a replay buffer prioritized by recency. As episodes (or sub-episodes) are sampled from the environment they are placed in a replay buffer and a priority p(s 0:T) is given to a trajectory s 0:T equivalent to the current training step. Then, to sample a batch for training, B episodes are sampled from the replay buffer proportional to exponentiated priority exp{βp(s 0:T)} for some hyperparameter β ≥ 0.For the prior policy πθ, we use a lagged geometric mean of the parameters. At each training step, we updateθ ← αθ + (1 − α)θ. Thus on average our training scheme attempts to maximize entropy regularized expected reward while penalizing divergence from a policy roughly 1/(1 − α) training steps in the past. The use of a relative entropy regularizer as a penalty rather than a constraint introduces several difficulties. The hyperparameter λ must necessarily adapt to the distribution of rewards. Thus, λ must be tuned not only to each environment but also during training on a single environment, since the observed reward distribution changes as the agent's behavior policy improves. Using a constraint form of the regularizer is more desirable, and others have advocated its use in practice BID28 specifically to robustly allow larger updates during training. To this end, we propose to redirect the hyperparameter tuning from λ to. Specifically, we present a method which, given a desired hard constraint on the relative entropy defined by, approximates the equivalent penalty coefficient λ. This is a key novelty of our work and is distinct from previous attempts at automatically tuning a regularizing coefficient, which iteratively increase and decrease the coefficient based on observed training behavior BID31 BID11.We restrict our analysis to the undiscounted setting γ = 1 with entropy regularizer τ = 0. Additionally, we assume deterministic, finite-horizon environment dynamics. An additional assumption we make is that the expected KL-divergence over states is well-approximated by the KL-divergence starting from the unique initial state s 0. Although in our experiments these restrictive assumptions are not met, we still found our method to perform well for adapting λ during training. In this setting the optimal policy of FORMULA0 is proportional to exponentiated scaled reward. Specifically, for a full episode s 0:T = (s 0, a 0, r 0, . . ., s T −1, a T −1, r T −1, s T), we have DISPLAYFORM0 where π(s 0: DISPLAYFORM1 We would like to approximate the trajectory-wide KL-divergence between π * andπ. We may express the KL-divergence analytically: DISPLAYFORM2 DISPLAYFORM3 Since all expectations are with respect toπ, this quantity is tractable to approximate given episodes sampled fromπ Therefore, in Trust-PCL, given a set of episodes sampled from the prior policy πθ and a desired maximum divergence, we can perform a simple line search to find a suitable λ which yields KL(π * ||πθ) as close as possible to.The preceding analysis provided a method to determine λ given a desired maximum divergence. However, there is still a question of whether should change during training. Indeed, as episodes may possibly increase in length, KL(π * ||π) naturally increases when compared to the average perstate KL(π * (−|s)||π(−|s)), and vice versa for decreasing length. Thus, in practice, given an and a set of sampled episodes S = {s DISPLAYFORM4, we approximate the best λ which yields a maximum divergence of N N k=1 T k . This makes it so that corresponds more to a constraint on the lengthaveraged KL-divergence. To avoid incurring a prohibitively large number of interactions with the environment for each parameter update, in practice we use the last 100 episodes as the set of sampled episodes S. While this is not exactly the same as sampling episodes from πθ, it is not too far off since πθ is a lagged version of the online policy π θ . Moreover, we observed this protocol to work well in practice. A more sophisticated and accurate protocol may be derived by weighting the episodes according to the importance weights corresponding to their true sampling distribution. We evaluate Trust-PCL against TRPO on a number of benchmark tasks. We choose TRPO as a baseline since it is a standard algorithm known to achieve state-of-the-art performance on the continuous control tasks we consider (see e.g., leaderboard on the OpenAI Gym website BID5). We find that Trust-PCL can match or improve upon TRPO's performance in terms of both average reward and sample efficiency. We chose a number of control tasks available from OpenAI Gym BID5. The first task, Acrobot, is a discrete-control task, while the remaining tasks (HalfCheetah, Swimmer, Hopper, Walker2d, and Ant) are well-known continuous-control tasks utilizing the MuJoCo environment BID33.For TRPO we trained using batches of Q = 25, 000 steps (12, 500 for Acrobot), which is the approximate batch size used by other implementations ). Thus, at each training iteration, TRPO samples 25, 000 steps using the policy πθ and then takes a single step within a KL-ball to yield a new π θ.Trust-PCL is off-policy, so to evaluate its performance we alternate between collecting experience and training on batches of experience sampled from the replay buffer. Specifically, we alternate between collecting P = 10 steps from the environment and performing a single gradient step based on a batch of size Q = 64 sub-episodes of length P from the replay buffer, with a recency weight of β = 0.001 on the sampling distribution of the replay buffer. To maintain stability we use α = 0.99 and we modified the loss from squared loss to Huber loss on the consistency error. Since our policy is parameterized by a unimodal Gaussian, it is impossible for it to satisfy all path consistencies, and so we found this crucial for stability. For each of the variants and for each environment, we performed a hyperparameter search to find the best hyperparameters. The plots presented here show the reward achieved during training on the best hyperparameters averaged over the best 4 seeds of 5 randomly seeded training runs. Note that this reward is based on greedy actions (rather than random sampling).Experiments were performed using Tensorflow BID0. Although each training step of Trust-PCL (a simple gradient step) is considerably faster than TRPO, we found that this does not have an overall effect on the run time of our implementation, due to a combination of the fact that each environment step is used in multiple training steps of Trust-PCL and that a majority of the run time is spent interacting with the environment. A detailed description of our implementation and hyperparameter search is available in the Appendix. We present the reward over training of Trust-PCL and TRPO in FIG0. We find that Trust-PCL can match or beat the performance of TRPO across all environments in terms of both final reward and sample efficiency. These are especially significant on the harder tasks (Walker2d and Ant). We additionally present our compared to other published in Table 1. We find that even when comparing across different implementations, Trust-PCL can match or beat the state-of-the-art. The most important hyperparameter in our method is, which determines the size of the trust region and thus has a critical role in the stability of the algorithm. To showcase this effect, we present the reward during training for several different values of in FIG1. As increases, instability increases as well, eventually having an adverse effect on the agent's ability to achieve optimal reward. The of Trust-PCL against a TRPO baseline. Each plot shows average greedy reward with single standard deviation error intervals capped at the min and max across 4 best of 5 randomly seeded training runs after choosing best hyperparameters. The x-axis shows millions of environment steps. We observe that Trust-PCL is consistently able to match and, in many cases, beat TRPO's performance both in terms of reward and sample efficiency. The of Trust-PCL across several values of, defining the size of the trust region. Each plot shows average greedy reward across 4 best of 5 randomly seeded training runs after choosing best hyperparameters. The x-axis shows millions of environment steps. We observe that instability increases with, thus concluding that the use of trust region is crucial. Note that standard PCL BID21 corresponds to → ∞ (that is, λ = 0). Therefore, standard PCL would fail in these environments, and the use of trust region is crucial. The main advantage of Trust-PCL over existing trust region methods for continuous control is its ability to learn in an off-policy manner. The degree to which Trust-PCL is off-policy is determined by a combination of the hyparparameters α, β, and P. To evaluate the importance of training off-policy, we evaluate Trust-PCL with a hyperparameter setting that is more on-policy. We set α = 0.95, β = 0.1, and P = 1, 000. In this setting, we also use large batches of Q = 25 episodes of length P (a total of 25, 000 environment steps per batch). Figure 3 shows the of Trust-PCL with our original parameters and this new setting. We note a dramatic advantage in sample efficiency when using off-policy training. Although Trust-PCL (on-policy) can achieve state-of-the-art reward performance, it requires an exorbitant amount of experience. On the other hand, Trust-PCL (off- Trust-PCL (on-policy) Trust-PCL (off-policy) Figure 3: The of Trust-PCL varying the degree of on/off-policy. We see that Trust-PCL (on-policy) has a behavior similar to TRPO, achieving good final reward but requiring an exorbitant number of experience collection. When collecting less experience per training step in Trust-PCL (off-policy), we are able to improve sample efficiency while still achieving a competitive final reward. BID9. These are each on different setups with different hyperparameter searches and in some cases different evaluation protocols (e.g.,TRPO (rllab) and IPG were run with a simple linear value network instead of the two-hidden layer network we use). Thus, it is not possible to make any definitive claims based on this data. However, we do conclude that our are overall competitive with state-of-the-art external implementations.policy) can be competitive in terms of reward while providing a significant improvement in sample efficiency. One last hyperparameter is τ, determining the degree of exploration. Anecdotally, we found τ to not be of high importance for the tasks we evaluated. Indeed many of our best use τ = 0. Including τ > 0 had a marginal effect, at best. The reason for this is likely due to the tasks themselves. Indeed, other works which focus on exploration in continuous control have found the need to propose exploration-advanageous variants of these standard benchmarks BID10. We have presented Trust-PCL, an off-policy algorithm employing a relative-entropy penalty to impose a trust region on a maximum reward objective. We found that Trust-PCL can perform well on a set of standard control tasks, improving upon TRPO both in terms of average reward and sample efficiency. Our best on Trust-PCL are able to maintain the stability and solution quality of TRPO while approaching the sample-efficiency of value-based methods (see e.g., BID18). This gives hope that the goal of achieving both stability and sample-efficiency without trading-off one for the other is attainable in a single unifying RL algorithm. We thank Matthew Johnson, Luke Metz, Shane Gu, and the Google Brain team for insightful comments and discussions. We have already highlighted the ability of Trust-PCL to use off-policy data to stably train both a parameterized policy and value estimate, which sets it apart from previous methods. We have also noted the ease with which exploration can be incorporated through the entropy regularizer. We elaborate on several additional benefits of Trust-PCL.Compared to TRPO, Trust-PCL is much easier to implement. Standard TRPO implementations perform second-order gradient calculations on the KL-divergence to construct a Fisher information matrix (more specifically a vector product with the inverse Fisher information matrix). This yields a vector direction for which a line search is subsequently employed to find the optimal step. Compare this to Trust-PCL which employs simple gradient descent. This makes implementation much more straightforward and easily realizable within standard deep learning frameworks. Even if one replaces the constraint on the average KL-divergence of TRPO with a simple regularization penalty (as in proximal policy gradient methods BID31 BID35), optimizing the ing objective requires computing the gradient of the KL-divergence. In Trust-PCL, there is no such necessity. The per-state KL-divergence need not have an analytically computable gradient. In fact, the KL-divergence need not have a closed form at all. The only requirement of Trust-PCL is that the log-density be analytically computable. This opens up the possible policy parameterizations to a much wider class of functions. While continuous control has traditionally used policies parameterized by unimodal Gaussians, with Trust-PCL the policy can be replaced with something much more expressive-for example, mixtures of Gaussians or autoregressive policies as in BID18.We have yet to fully explore these additional benefits in this work, but we hope that future investigations can exploit the flexibility and ease of implementation of Trust-PCL to further the progress of RL in continuous control environments. We describe in detail the experimental setup regarding implementation and hyperparameter search. In Acrobot, episodes were cut-off at step 500. For the remaining environments, episodes were cutoff at step 1, 000.Acrobot, HalfCheetah, and Swimmer are all non-terminating environments. Thus, for these environments, each episode had equal length and each batch contained the same number of episodes. Hopper, Walker2d, and Ant are environments that can terminate the agent. Thus, for these environments, the batch size throughout training remained constant in terms of steps but not in terms of episodes. There exists an additional common MuJoCo task called Humanoid. We found that neither our implementation of TRPO nor Trust-PCL could make more than negligible headway on this task, and so omit it from the . We are aware that TRPO with the addition of GAE and enough finetuning can be made to achieve good on Humanoid. We decided to not pursue a GAE implementation to keep a fair comparison between variants. Trust-PCL can also be made to incorporate an analogue to GAE (by maintaining consistencies at varying time scales), but we leave this to future work. We use fully-connected feed-forward neural networks to represent both policy and value. The policy π θ is represented by a neural network with two hidden layers of dimension 64 with tanh activations. At time step t, the network is given the observation s t. It produces a vector µ t, which is combined with a learnable (but t-agnostic) parameter ξ to parametrize a unimodal Gaussian with mean µ t and standard deviation exp(ξ). The next action a t is sampled randomly from this Gaussian. The value network V φ is represented by a neural network with two hidden layers of dimension 64 with tanh activations. At time step t the network is given the observation s t and the component-wise squared observation s t s t. It produces a single scalar value. At each training iteration, both the policy and value parameters are updated. The policy is trained by performing a trust region step according to the procedure described in BID28.The value parameters at each step are solved using an LBFGS optimizer. To avoid instability, the value parameters are solved to fit a mixture of the empirical values and the expected values. That is, we determine φ to minimize s∈batch (V φ (s) − κVφ(s) − (1 − κ)Vφ(s)) 2, where againφ is the previous value parameterization. We use κ = 0.9. This method for training φ is according to that used in. At each training iteration, both the policy and value parameters are updated. The specific updates are slightly different between Trust-PCL (on-policy) and Trust-PCL (off-policy).For Trust-PCL (on-policy), the policy is trained by taking a single gradient step using the Adam optimizer BID14 with learning rate 0.001. The value network update is inspired by that used in TRPO we perform 5 gradients steps with learning rate 0.001, calculated with regards to a mix between the empirical values and the expected values according to the previousφ. We use κ = 0.95.For Trust-PCL (off-policy), both the policy and value parameters are updated in a single step using the Adam optimizer with learning rate 0.0001. For this variant, we also utilize a target value network (lagged at the same rate as the target policy network) to replace the value estimate at the final state for each path. We do not mix between empirical and expected values. We found the most crucial hyperparameters for effective learning in both TRPO and Trust-PCL to be (the constraint defining the size of the trust region) and d (the rollout determining how to evaluate the empirical value of a state). For TRPO we performed a grid search over ∈ {0.01, 0.02, 0.05, 0.1}, d ∈ {10, 50}. For Trust-PCL we performed a grid search over ∈ {0.001, 0.002, 0.005, 0.01}, d ∈ {10, 50}. For Trust-PCL we also experimented with the value of τ, either keeping it at a constant 0 (thus, no exploration) or decaying it from 0.1 to 0.0 by a smoothed exponential rate of 0.1 every 2,500 training iterations. We fix the discount to γ = 0.995 for all environments. A simplified pseudocode for Trust-PCL is presented in Algorithm 1. Input: Environment EN V, trust region constraint, learning rates η π, η v, discount factor γ, rollout d, batch size Q, collect steps per train step P, number of training steps N, replay buffer RB with exponential lag β, lag on prior policy α.function Gradients({s // Update auxiliary variables Updateθ = αθ + (1 − α)θ. Update λ in terms of according to Section 4.3. end for

Represent the short summary for finding the corresponding scientific article (long)

We extend recent insights related to softmax consistency to achieve state-of-the-art results in continuous control.

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.