Question: Represent the scientific article to find a short TLDR

Question: As for knowledge-based question answering, a fundamental problem is to relax the assumption of answerable questions from simple questions to compound questions. Traditional approaches firstly detect topic entity mentioned in questions, then traverse the knowledge graph to find relations as a multi-hop path to answers, while we propose a novel approach to leverage simple-question answerers to answer compound questions. Our model consists of two parts: (i) a novel learning-to-decompose agent that learns a policy to decompose a compound question into simple questions and (ii) three independent simple-question answerers that classify the corresponding relations for each simple question. Experiments demonstrate that our model learns complex rules of compositionality as stochastic policy, which benefits simple neural networks to achieve state-of-the-art on WebQuestions and MetaQA. We analyze the interpretable decomposition process as well as generated partitions. Knowledge-Based Question Answering (KBQA) is one of the most interesting approaches of answering a question, which bridges a curated knowledge base of tremendous facts to answerable questions. With question answering as a user-friendly interface, users can easily query a knowledge base through natural language, i.e., in their own words. In the past few years, many systems BID5 BID2; BID11 BID13 have achieved remarkable improvements in various datasets, such as WebQuestions BID5, SimpleQuestions BID6 and MetaQA.However, most of them BID31 BID6 BID10 BID34 BID36 assume that only simple questions are answerable. Simple questions are questions that have only one relation from the topic entity to unknown tail entities (answers, usually substituted by an interrogative word) while compound questions are questions that have multiple 1 relations. For example, "Who are the daughters of Barack Obama?" is a simple question and "Who is the mother of the daughters of Barack Obama?" is a compound question which can be decomposed into two simple questions. In this paper, we aim to relax the assumption of answerable questions from simple questions to compound questions. Figure 1 illustrates the process of answering compound questions. Intuitively, to answer a compound question, traditional approaches firstly detect topic entity mentioned in the question, as the starting point for traversing the knowledge graph, then find a chain of multiple (≤ 3) relations as a multi-hop 2 path to golden answers. We propose a learning-to-decompose agent which assists simple-question answerers to solve compound questions directly. Our agent learns a policy for decomposing compound question into simple ones in a meaningful way, guided by the feedback from the downstream simple-question answerers. The goal of the agent is to produce partitions and compute the compositional structure of questions 1 We assume that the number of corresponding relations is at most three. 2 We are aware of the term multi-hop question in the literature. We argue that compound question is a better fit for the context of KBQA since multi-hop characterizes a path, not a question. As for document-based QA, multi-hop also refers to routing over multiple evidence to answers. Figure 1: An example of answering compound questions. Given a question Q, we first identify the topic entity e with entity linking. By relation detection, a movie-to-actor relation f 1, an actor-tomovie relation f 2 and a movie-to-writer relation f 3 forms a path to the answers W i. Note that each relation f i corresponds to a part of the question. If we decomposes the question in a different way, we may find a movie-to-movie relation g as a shortcut, and g(e) = f 2 (f 1 (e)) = (f 2 • f 1)(e) holds. Our model discovered such composite rules. See section 4 for further discussion.with maximum information utilization. The intuition is that encouraging the model to learn structural compositions of compound questions will bias the model toward better generalizations about how the meaning of a question is encoded in terms of compositional structures on sequences of words, leading to better performance on downstream question answering tasks. We demonstrate that our agent captures the semantics of compound questions and generate interpretable decomposition. Experimental show that our novel approach achieves state-of-the-art performance in two challenging datasets (WebQuestions and MetaQA), without re-designing complex neural networks to answer compound questions. For combinational generalization BID4 on the search space of knowledge graph, many approaches BID31 BID34 tackle KBQA in a tandem manner, i.e., topic entity linking followed by relation detection. An important line of research focused on directly parsing the semantics of natural language questions to structured queries BID8 BID17 BID2 BID31. An intermediate meaning representation or logical form is generated for query construction. It often requires pre-defined rules or grammars BID5 ) based on hand-crafted features. By contrast, another line of research puts more emphasis on representing natural language questions instead of constructing knowledge graph queries. Employing CNNs BID11 BID34 or RNNs BID10 BID36, variable-length questions are compressed into their corresponding fix-length vector. Most approaches in this line focus on solving simple questions because of the limited expression power of fix-length vector, consistent with observations BID23 BID0 in Seq2Seq task such as Neural Machine Translation. Closely related to the second line of research, our proposed model learns to decompose compound question into simple questions, which eases the burden of learning vector representations for compound question. Once the decomposition process is completed, a simple-question answerer directly decodes the vector representation of simple questions to an inference chain of relations with the desired order, which resolves the bottleneck of KBQA. Many reinforcement learning approaches learn sentence representations in a bottom-up manner. BID35 learn tree structures for the order of composing words into sentences using reinforcement learning with Tree-LSTM BID24 BID40, while BID37 employ REINFORCE BID28 to select useful words sequentially. Either in tree structure or sequence, the vector representation is built up from the words, which benefits the downstream natural language processing task such as text classification BID22 and natural language inference BID7. By contrast, from the top down, our proposed model learns to decompose compound questions into simple questions, which helps to tackle the bottleneck of KBQA piece by piece. See section 3 for more details. Natural question understanding has attracted the attention of different communities. BID15 introduce SequentialQA task that requires to parse the text to SQL which locates table cells as answers. The questions in SequentialQA are decomposed from selected questions of WikiTableQuestions dataset BID19 by crowdsourced workers while we train an agent to decompose questions using reinforcement learning. BID25 propose a ComplexWebQuestion dataset that contains questions with compositional semantics while BID3 collects a dataset called ComplexQuestions focusing on multi-constrained knowledge-based question answering. The closest idea to our work is BID25 which adopts a pointer network to decompose questions and a reading comprehension model to retrieve answers from the Web. The main difference is that they leverage explicit supervisions to guide the pointer network to correctly decompose complex web questions based on human logic (e.g., conjunction or composition) while we allow the learning-to-decompose agent to discover good partition strategies that benefit downstream task. Note that it is not necessarily consistent with human intuition or linguistic knowledge. Without heavy feature engineering, semantic role labeling based on deep neural networks BID9 focus on capturing dependencies between predicates and arguments by learning to label semantic roles for each word. build an end-to-end system which takes only original text information as input features, showing that deep neural networks can outperform traditional approaches by a large margin without using any syntactic knowledge. BID18 improve the role classifier by incorporating vector representations of both input sentences and predicates. BID26 handle structural information and long-range dependencies with self-attention mechanism. This line of work concentrates on improving role classifier. It still requires rich supervisions for training the role classifier at the token level. Our approach also requires to label an action for each word, which is similar to role labeling. However, we train our approach at the sentence level which omits word-by-word annotations. Our learning-to-decompose agent generates such annotations on the fly by exploring the search space of strategies and increases the probability of good annotations according to the feedback. FIG0 illustrates an overview of our model and the data flow. Our model consists of two parts: a learning-to-decompose agent that decomposes each input question into at most three partitions and three identical simple-question answerers that map each partition to its corresponding relation independently. We refer to the learning-to-decompose agent as the agent and three simple-question answerers as the answerers in the rest of our paper for simplicity. Our main idea is to best divide an input question into at most three partitions which each partition contains the necessary information for the downstream simple-question answerer. Given an input A zoom-in version of the lower half of figure 2. Our agent consists of two components: a Memory Unit and an Action Unit. The Memory Unit observes current word at each time step t and updates the state of its own memory. We use a feedforward neural network as policy network for the Action Unit. question of N words 3 x = {x 1, x 2, . . ., x N}, we assume that a sequence of words is essentially a partially observable environment and we can only observe the corresponding vector representation o t = x t ∈ R D at time step t. FIG1 summarizes the process for generating decision of compound question decomposition. The agent has a Long Short-Term Memory (LSTM; BID14) cell unrolling for each time step to memorize input history. DISPLAYFORM0 where The state s t ∈ R 2H of the agent is defined as DISPLAYFORM1 DISPLAYFORM2 which maintained by the above memory cell (Eq. 1) unrolling for each time step. [·, ·] denotes the concatenation of two vectors. Action Unit The agent also has a stochastic policy network π(α|s; W π) where W π is the parameter of the network. Specifically, we use a two-layer feedforward network that takes the agent's state s as its input: DISPLAYFORM3 where W DISPLAYFORM4 π ∈ R 3×H and b DISPLAYFORM5 Following the learned policy, the agent decomposes a question of length N by generating a sequence of actions α t ∈ {1st, 2nd, 3rd}, t = 1, 2,..., N. Words under the same decision (e.g. 1st) will be appended into the same sub-sequence (e.g. the first partition).Formally, DISPLAYFORM6 denotes the partitions of a question. Note that in a partition, words are not necessarily consecutive 4. The relative position of two words in original question is preserved. t 1 + t 2 + t 3 = N holds for every question. Reward The episodic reward R will be +1 if the agent helps all the answerers to get the golden answers after each episode, or −1 otherwise. There is another reward function R = Σ log P (Y * | X) that is widely used in the literature of using reinforcement learning for natural language processing task BID1 BID37. We choose the former as reward function for lower variance. Each unique rollout (sequence of actions) corresponds to unique compound question decomposition. We do not assume that any question should be divided into exactly three parts. We allow our agent to explore the search space of partition strategies and to increase the probability of good ones. The goal of our agent is to learn partition strategies that benefits the answerers the most. With the help of the learning-to-decompose agent, simple-question answerers can answer compound questions. Once the question is decomposed into partitions as simple questions, each answerer takes its partition DISPLAYFORM0 } as input and classifies it as the corresponding relation in knowledge graph. For each partition x (k), we use LSTM network to construct simple-question representation directly. The partition embedding is the last hidden state of LSTM network, denoted by x (k) ∈ R 2H. We again use a two-layer feedforward neural network to make prediction, i.e. estimate the likelihood of golden relation r. DISPLAYFORM1 where DISPLAYFORM2 C is the number of classes. Each answerer only processes its corresponding partition and outputs a predicted relation. These three modules share no parameters except the embedding layer because our agent will generates conflict assignments for the same questions in different epoches. If all the answerers share same parameters in different layers, data conflicts undermine the decision boundary and leads to unstable training. Note that we use a classification network for sequential inputs that is as simple as possible. In addition to facilitating the subsequent theoretical analysis, the simple-question answerers we proposed are much simpler than good baselines for simple question answering over knowledge graph, without modern architecture features such as bi-directional process, read-write memory BID6, attention mechanism BID34 or residual connection BID36.The main reason is that our agent learns to decompose input compound questions to the simplified version which is answerable for such simple classifiers. This can be a strong evidence for validating the agent's ability on compound question decomposition. The agent and the answerers share the same embeddings. The agent can only observe word embeddings while the answerers are allowed to update them in the backward pass. We train three simple-question answerers separately using Cross Entropy loss between the predicted relation and the golden relation. These three answerers are independent of each other. We do not use the pre-train trick for all the experiments since we have already observed consistent convergence on different task settings. We reduce the variance of Monte-Carlo Policy Gradient estimator by taking multiple (≤ 5) rollouts for each question and subtracting a baseline that estimates the expected future reward given the observation at each time step. The Baseline We follow Ranzato et al. FORMULA0 which uses a linear regressor which takes the agent's memory state s t as input and minimizes the mean squared loss for training. Such a loss signal is used for updating the parameters of baseline only. The regressor is an unbiased estimator of expected future rewards since it only depends on the agent's memory states. Our agent learns a optimal policy to decompose compound questions into simple ones using MonteCarlo Policy Gradient (MCPG) method. The partitions of question is then feeded to corresponding simple-question answerers for policy evaluation. The agent takes the final episodic reward in return. The goal of our experiments is to evaluate our hypothesis that our model discovers useful question partitions and composition orders that benefit simple-question answerers to tackle compound question answering. Our experiments are three-fold. First, we trained the proposed model to master the order of arithmetic operators (e.g., + − ×÷) on an artificial dataset. Second, we evaluate our method on the standard benchmark dataset MetaQA. Finally, we discuss some interesting properties of our agent by case study. The agent's ability of compound question decomposition can be viewed as the ability of priority assignment. To validate the decomposition ability of our proposed model, we train our model to master the order of arithmetic operations. We generate an artificial dataset of complex algebraic expressions. (e.g. 1 + 2 − 3 × 4 ÷ 5 =? or 1 + (2 − 3) × 4 ÷ 5). The algebraic expression is essentially a question in math language which the corresponding answer is simply a real number. Specifically, the complex algebraic expression is a sequence of arithmetic operators including +, −, ×, ÷, (and). We randomly sample a symbol sequence of length N, with restriction of the legality of parentheses. The number of parentheses is P (≤ 2). The number of symbols surrounding by parentheses is Q. The position of parentheses is randomly selected to increase the diversity of expression patterns. For example, (+×)+(÷) and +×(+×)−÷ are data points (1+2×3)+(4÷5) and 1 + 2 × (3 + 4 × 5) − 6 ÷ 7 with N = 8.This task aims to test the learning-to-decompose agent whether it can assign a feasible order of arithmetic operations. We require the agent to assign higher priority for operations surrounding by parentheses and lower priority for the rest of operations. We also require that our agent can learn a policy from short expressions (N ≤ 8), which generalizes to long ones (13 ≤ N ≤ 16).We use 100-dimensional (D = 100) embeddings for symbols with Glorot initialization BID12. The dimension of hidden state and cell state of memory unit H is 128. We use the RMSProp optimizer BID27 to train all the networks with the parameters recommended in the original paper except the learning rate α. The learning rate for the agent and the answerers is 0.00001 while the learning rate for the baseline is 0.0001. We test the performance in different settings. TAB0 summarizes the experiment . DISPLAYFORM0 99.21 N = 8, P = 1, Q = 3 N = 13, P = 1, Q = 3 93.37 N = 8, P = 1, Q = 3 N = 13, P = 1, Q = 7 66.42The first line indicates that our agent learns an arithmetic skill that multiplication and division have higher priority than addition and subtraction. The second line indicates that our agent learns to discover the higher-priority expression between parentheses. The third line, compared to the second line, indicates that increasing the distance between two parentheses could harm the performance. We argue that this is because of the Long Short-Term Memory Unit of our agent suffers when carrying the information of left parenthesis for such a long distance. We evaluate our proposed model on the test set of two challenging KBQA research dataset, i.e., WebQuestions BID5 and MetaQA. Each question in both datasets is labeled with the golden topic entity and the inference chain of relations. The statistics of MetaQA dataset is shown in table 2. The number of compound questions in MetaQA is roughly twice that of simple questions. The max length of training questions is 16. The size of vocabulary in questions is 39,568. The coupled knowledge graph contains 43,234 entities and 9 relations. We also augmented the relation set with the inversed relations, as well as a "NO OP" relation as placeholder. The total number of relations we used is 14 since some inversed relations are meaningless. WebQuestions contains 2,834 questions for training, 944 questions for validation and 2,032 questions for testing respectively. We use 602 relations for the relation classification task. The number of compound questions in WebQuestions is roughly equal to that of simple questions. Note that a compound question in WebQuestions is decomposed into two partitions since the maximum number of corresponding relations is two. One can either assume topic entity of each question is linked or use a simple string matching heuristic like character trigrams matching to link topic entity to knowledge graph directly. We use the former setting while the performance of the latter is reasonable good. We tend to evaluate the relation detection performance directly. For both datasets, we use 100-dimensional (D = 100) word embeddings with Glorot initialization BID12. The dimension of hidden state and cell state of memory unit H is 128. We use the RMSProp optimizer BID27 to train the agent with the parameters recommended in the original paper except the learning rate. We train the rest of our model using Adam BID16 with default parameters. The learning rate for all the modules is 0.0001 no matter the optimizer it is. We use four samples for Monte-Carlo Policy Gradient estimator of REINFORCE. The metric for relation detection is overall accuracy that only cumulates itself if all relations of a compound question are correct. TAB2 presents our on MetaQA dataset. The last column for total accuracy is the most representative for our model's performance since the only assumption we made about input questions is the number of corresponding relations is at most three. BID36 on this dataset focus on leveraging information of the name of Freebase relation while we are only using question information for classification. We assume that the compound question can be decomposed into at most three simple questions. In practice, this generalized assumption of answerable questions is not necessary. One example is that WebQuestions only contains compound questions corresponding to two but not three relations. It indicates that people tend to ask less complicated questions more often. So we conduct an ablation study for the hyperparameters of this central assumption in our paper. We assume that all the questions in MetaQA dataset contain at most two corresponding relations. We run the same code with the same hyperparameters except we only use two simple-question answerers. The purpose of the evaluation is to prove that our model improves performance on 1-hop and 2-hop questions by giving up the ability to answer three-hop questions. TAB4 presents our on ablation test. We can draw a that there exists a trade-off between answering more complex questions and achieving better performance by limiting the size of search space. Figure 4 illustrates a continuous example of figure 1 for the case study, which is generated by our learning-to-decompose agent. Assuming the topic entity e is detected and replaced by a placeholder, the agent may discover two different structures of the question that is consistent with human intuition. Since the knowledge graph does not have a movie-to-movie relation named "share actor with", the lower partition can not help the answerers classify relations correctly. However, the upper partition will be rewarded. As a , our agent optimizes its strategies such that it can decompose the original question in the way that benefits the downstream answerers the most. We observe the fact that our model understands the concept of "share" as the behavior "take the inversed relation". That is, "share actors" in a question is decomposed to "share" and "actors" in two partitions. The corresponding formulation is g(e) = f 2 (f 1 (e)) = (f 2 • f 1)(e). We observe the same phenomenon on "share directors". We believe it is a set of strong evidence for supporting our main claims. Understanding compound questions, in terms of The Principle of Semantic Compositionality BID20, require one to decompose the meaning of a whole into the meaning of parts. While previous works focus on leveraging knowledge graph for generating a feasible path to answers, we Figure 4: A continuous example of figure 1. The hollow circle indicates the corresponding action the agent takes for each time step. The upper half is the actual prediction while the lower half is a potential partition. Since we do not allow a word to join two partitions, the agent learns to separate "share" and "actors" into different partitions to maximize information utilization.propose a novel approach making full use of question semantics efficiently, in terms of the Principle of Semantic Compositionality. In other words, it is counterintuitive that compressing the whole meaning of a variable-length sentence to a fixed-length vector, which leaves the burden to the downstream relation classifier. In contrast, we assume that a compound question can be decomposed into three simple questions at most. Our model generates partitions by a learned policy given a question. The vector representations of each partition are then fed into the downstream relation classifier. While previous works focus on leveraging knowledge graph for generating a feasible path to answers, we propose a novel approach making full use of question semantics efficiently, in terms of the Principle of Semantic Compositionality. Our learning-to-decompose agent can also serve as a plug-and-play module for other question answering task that requires to understand compound questions. This paper is an example of how to help the simple-question answerers to understand compound questions. The answerable question assumption must be relaxed in order to generalize question answering.

Represent the TLDR for finding the corresponding scientific article

We propose a learning-to-decompose agent that helps simple-question answerers to answer compound question over knowledge graph.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the scientific article to find a short TLDR

Question: Energy based models outputs unmormalized log-probability values given datasamples. Such a estimation is essential in a variety of application problems suchas sample generation, denoising, sample restoration, outlier detection, Bayesianreasoning, and many more. However, standard maximum likelihood training iscomputationally expensive due to the requirement of sampling model distribution. Score matching potentially alleviates this problem, and denoising score matching is a particular convenient version. However, previous attemptsfailed to produce models capable of high quality sample synthesis. We believethat it is because they only performed denoising score matching over a singlenoise scale. To overcome this limitation, here we instead learn an energy functionusing all noise scales. When sampled using Annealed Langevin dynamics andsingle step denoising jump, our model produced high-quality samples comparableto state-of-the-art techniques such as GANs, in addition to assigning likelihood totest data comparable to previous likelihood models. Our model set a new sam-ple quality baseline in likelihood-based models. We further demonstrate that our model learns sample distribution and generalize well on an image inpainting tasks. Treating data as stochastic samples from a probability distribution and developing models that can learn such distributions is at the core for solving a large variety of application problems, such as error correction/denoising , outlier/novelty detection , sample generation , invariant pattern recognition, Bayesian reasoning which relies on good data priors, and many others. Energy-Based Models (EBMs) assign an energy E(x x x) to each data point x x x which implicitly defines a probability by the Boltzmann distribution p m (x x x) = e −E(x x x) /Z. Sampling from this distribution can be used as a generative process that yield plausible samples of x x x. Compared to other generative models, like GANs , flowbased models , or auto-regressive models (van den ;), energy-based models have significant advantages. First, they provide explicit (unnormalized) density information, compositionality , better mode coverage and flexibility . Further, they do not require special model architecture, unlike auto-regressive and flow-based models. Recently, Energy-based models has been successfully trained with maximum likelihood , but training can be very computationally demanding due to the need of sampling model distribution. Variants with a truncated sampling procedure have been proposed, such as contrastive divergence . Such models learn much faster with the draw back of not exploring the state space thoroughly . Score matching (SM) (Hyvärinen, 2005) circumvents the requirement of sampling the model distribution. In score matching, the score function is defined to be the gradient of log-density or the negative energy function. The expected L2 norm of difference between the model score function and the data score function are minimized. One convenient way of using score matching is learning the energy function corresponding to a Gaussian kernel Parzen density estimator of the data: p σ0 (x x x) = q σ0 (x x x|x x x)p(x x x)dx x x. Though hard to evaluate, the data score is well defined: s d (x x x) = ∇x x x log(p σ0 (x x x)), and the corresponding objective is: L SM (θ) = E pσ0(x x x) ∇x x x log(p σ0 (x x x)) + ∇x x x E(x x x; θ) studied the connection between denoising auto-encoder and score matching, and proved the remarkable that the following objective, named Denoising Score Matching (DSM), is equivalent to the objective above: L DSM (θ) = E pσ 0 (x x x,x x x) ∇x x x log(q σ0 (x x x|x x x)) + ∇x x x E(x x x; θ) Note that in the Parzen density score is replaced by the derivative of log density of the single noise kernel ∇x x x log(q σ0 (x x x|x x x)), which is much easier to evaluate. In the particular case of Gaussian noise, log(q σ0 (x x x|x x x)) = − (x x x−x x x) 2 2σ 2 0 + C, and therefore: The interpretation of objective is simple, it forces the energy gradient to align with the vector pointing from the noisy sample to the clean data sample. To optimize an objective involving the derivative of a function defined by a neural network, proposed the use of double backpropagation . Deep energy estimator networks first applied this technique to learn an energy function defined by a deep neural network. In this work and similarly in , an energy-based model was trained to match a Parzen density estimator of data with a certain noise magnitude. The previous models were able to perform denoising task, but they were unable to generate high-quality data samples from a random input initialization. Recently, trained an excellent generative model by fitting a series of score estimators coupled together in a single neural network, each matching the score of a Parzen estimator with a different noise magnitude. The questions we address here is why learning energy-based models with single noise level does not permit high-quality sample generation and what can be done to improve energy based models. Our work builds on key ideas from;;. Section 2, provides a geometric view of the learning problem in denoising score matching and provides a theoretical explanation why training with one noise level is insufficient if the data dimension is high. Section 3 presents a novel method for training energy based model, Multiscale Denoising Score Matching (MDSM). Section 4 describes empirical of the MDSM model and comparisons with other models. 2 A GEOMETRIC VIEW OF DENOISING SCORE MATCHING used denoising score matching with a range of noise levels, achieving great empirical . The authors explained that large noise perturbation are required to enable the learning of the score in low-data density regions. But it is still unclear why a series of different noise levels are necessary, rather than one single large noise level. , we analyze the learning process in denoising score matching based on measure concentration properties of high-dimensional random vectors. We adopt the common assumption that the data distribution to be learned is high-dimensional, but only has support around a relatively low-dimensional manifold (; ;). If the assumption holds, it causes a problem for score matching: The density, or the gradient of the density is then undefined outside the manifold, making it difficult to train a valid density model for the data distribution defined on the entire space. and discussed this problem and proposed to smooth the data distribution with a Gaussian kernel to alleviate the issue. To further understand the learning in denoising score matching when the data lie on a manifold X and the data dimension is high, two elementary properties of random Gaussian vectors in highdimensional spaces are helpful: First, the length distribution of random vectors becomes concentrated at √ dσ , where σ 2 is the variance of a single dimension. Second, a random vector is always close to orthogonal to a fixed vector . With these premises one can visualize the configuration of noisy and noiseless data points that enter the learning process: A data point x x x sampled from X and its noisy versionx x x always lie on a line which is almost perpendicular to the tangent space T x x x X and intersects X at x x x. Further, the distance vectors between (x x x,x x x) pairs all have similar length √ dσ. As a consequence, the set of noisy data points concentrate on a setX √ that has a distance with (Therefore, performing denoising score matching learning with (x x x,x x x) pairs generated with a fixed noise level σ, which is the approach taken previously except in, will match the score in the setX √ dσ, and enable denoising of noisy points in the same set. However, the learning provides little information about the density outside this set, farther or closer to the data manifold, as noisy samples outsideX √ dσ, rarely appear in the training process. An illustration is presented in Figure 1A. ) is very small in high-dimensional space, the score inX still plays a critical role in sampling from random initialization. This analysis may explain why models based on denoising score matching, trained with a single noise level encounter difficulties in generating data samples when initialized at random. For an empirical support of this explanation, see our experiments with models trained with single noise magnitudes (Appendix B). To remedy this problem, one has to apply a learning procedure of the sort proposed in, in which samples with different noise levels are used. Depending on the dimension of the data, the different noise levels have to be spaced narrowly enough to avoid empty regions in the data space. In the following, we will use Gaussian noise and employ a Gaussian scale mixture to produce the noisy data samples for the training (for details, See Section 3.1 and Appendix A). Another interesting property of denoising score matching was suggested in the denoising autoencoder literature . With increasing noise level, the learned features tend to have larger spatial scale. In our experiment we observe similar phenomenon when training model with denoising score matching with single noise scale. If one compare samples in Figure B.1, Appendix B, it is evident that noise level of 0.3 produced a model that learned short range correlation that spans only a few pixels, noise level of 0.6 learns longer stroke structure without coherent overall structure, and noise level of 1 learns more coherent long range structure without details such as stroke width variations. This suggests that training with single noise level in denoising score matching is not sufficient for learning a model capable of high-quality sample synthesis, as such a model have to capture data structure of all scales. Motivated by the analysis in section 2, we strive to develop an EBM based on denoising score matching that can be trained with noisy samples in which the noise level is not fixed but drawn from a distribution. The model should approximate the Parzen density estimator of the data p σ0 (x x x) = q σ0 (x x x|x x x)p(x x x)dx. Specifically, the learning should minimize the difference between the derivative of the energy and the score of p σ0 under the expectation E p M (x x x) rather than E pσ 0 (x x x), the expectation taken in standard denoising score matching. Here p M (x x x) = q M (x x x|x x x)p(x x x)dx is chosen to cover the signal space more evenly to avoid the measure concentration issue described above. The ing Multiscale Score Matching (MSM) objective is: Compared to the objective of denoising score matching, the only change in the new objective is the expectation. Both objectives are consistent, if p M (x x x) and p σ0 (x x x) have the same support, as shown formally in Proposition 1 of Appendix A. In Proposition 2, we prove that Equation 4 is equivalent to the following denoising score matching objective: The above hold for any noise kernel q σ0 (x x x|x x x), but Equation 5 contains the reversed expectation, which is difficult to evaluate in general. To proceed, we choose q σ0 (x x x|x x x) to be Gaussian, and also choose q M (x x x|x x x) to be a Gaussian scale mixture: q M (x x x|x x x) = q σ (x x x|x x x)p(σ)dσ and q σ (x x x|x x x) = N (x x x, σ 2 I d). After algebraic manipulation and one approximation (see the derivation following Proposition 2 in Appendix A), we can transform Equation 5 into a more convenient form, which we call Multiscale Denoising Score Matching (MDSM): The square loss term evaluated at noisy pointsx x x at larger distances from the true data points x x x will have larger magnitude. Therefore, in practice it is convenient to add a monotonically decreasing term l(σ) for balancing the different noise scales, e.g. l(σ) = 1 σ 2. Ideally, we want our model to learn the correct gradient everywhere, so we would need to add noise of all levels. However, learning denoising score matching at very large or very small noise levels is useless. At very large noise levels the information of the original sample is completely lost. Conversely, in the limit of small noise, the noisy sample is virtually indistinguishable from real data. In neither case one can learn a gradient which is informative about the data structure. Thus, the noise range needs only to be broad enough to encourage learning of data features over all scales. Particularly, we do not sample σ but instead choose a series of fixed σ values, we arrive at the final objective: It may seem that σ 0 is an important hyperparameter to our model, but after our approximation σ 0 become just a scaling factor in front of the energy function, and can be simply set to one as long as the temperature range during sampling is scaled accordingly (See Section 3.2). Therefore the only hyper-parameter is the rang of noise levels used during training. On the surface, objective looks similar to the one in. The important difference is that Equation 7 approximates a single distribution, namely p σ0 (x x x), the data smoothed with one fixed kernel q σ0 (x x x|x x x). In contrast, approximate the score of multiple distributions, the family of distributions {p σi (x x x): i = 1,..., n}, ing from the data smoothed by kernels of different widths σ i. Because our model learns only a single target distribution, it does not require noise magnitude as input. Langevin dynamics has been used to sample from neural network energy functions . However, the studies described difficulties with mode exploration unless very large number of sampling steps is used. To improve mode exploration, we propose incorporating simulated annealing in the Langevin dynamics. Simulated annealing improves mode exploration by sampling first at high temperature and then cooling down gradually. This has been successfully applied to challenging computational problems, such as combinatorial optimization. To apply simulated annealing to Langevin dynamics. Note that in a model of Brownian motion of a physical particle, the temperature in the Langevin equation enters as a factor √ T in front of the noise term, some literature uses β −1 where β = 1/T . Adopting the √ T convention, the Langevin sampling process is given by: where T t follows some annealing schedule, and denotes step length, which is fixed. During sampling, samples behave very much like physical particles under Brownian motion in a potential field. Because the particles have average energies close to the their current thermic energy, they explore the state space at different distances from data manifold depending on temperature. Eventually, they settle somewhere on the data manifold. The behavior of the particle's energy value during a typical annealing process is depicted in Appendix Figure F.1B. If the obtained sample is still slightly noisy, we can apply a single step gradient denoising jump to improve sample quality: This denoising procedure can be applied to noisy sample with any level of Gaussian noise because in our model the gradient automatically has the right magnitude to denoise the sample. This process is justified by the Empirical Bayes interpretation of this denoising process, as studied in. also call their sample generation process annealed Langevin dynamics. It should be noted that their sampling process does not coincide with Equation 8. Their sampling procedure is best understood as sequentially sampling a series of distributions corresponding to data distribution corrupted by different levels of noise. Training and Sampling Details. The proposed energy-based model is trained on standard image datasets, specifically MNIST, Fashion MNIST, CelebA and CIFAR-10 . During training we set σ 0 = 0.1 and train over a noise range of σ ∈ [0.05, 1.2], with the different noise uniformly spaced on the batch dimension. For MNIST and Fashion MNIST we used geometrically distributed noise in the range [0.1, 3]. The weighting factor l(σ) is always set to 1/σ 2 to make the square term roughly independent of σ. We fix the batch size at 128 and use the Adam optimizer with a learning rate of 5 × 10 −5. For MNIST and Fashion MNIST, we use a 12-Layer ResNet with 64 filters, for the CelebA and CIFAT-10 data sets we used a 18-Layer ResNet with 128 filters (a; . No normalization layer was used in any of the networks. We designed the output layer of all networks to take a generalized quadratic form . Because the energy function is anticipated to be approximately quadratic with respect to the noise level, this modification was able to boost the performance significantly. For more detail on training and model architecture, see Appendix D. One notable is that since our training method does not involve sampling, we achieved a speed up of roughly an order of magnitude compared to the maximum-likelihood training using Langevin dynamics 1. Our method thus enables the training of energy-based models even when limited computational resources prohibit maximum likelihood methods. We found that the choice of the maximum noise level has little effect on learning as long as it is large enough to encourage learning of the longest range features in the data. However, as expected, learning with too small or too large noise levels is not beneficial and can even destabilize the training process. Further, our method appeared to be relatively insensitive to how the noise levels are distributed over a chosen range. Geometrically spaced noise as in and linearly spaced noise both work, although in our case learning with linearly spaced noise was somewhat more robust. For sampling the learned energy function we used annealed Langevin dynamics with an empirically optimized annealing schedule,see Figure F.1B for the particular shape of annealing schedule we used. In contrast, annealing schedules with theoretical guaranteed convergence property takes extremely long . The range of temperatures to use in the sampling process depends on the choice of σ 0, as the equilibrium distribution is roughly images with Gaussian noise of magnitude √ T σ 0 added on top. To ease traveling between modes far apart and ensure even sampling, the initial temperature needs to be high enough to inject noise of sufficient magnitude. A choice of T = 100, which corresponds to added noise of magnitude √ 100 * 0.1 = 1, seems to be sufficient starting point. For step length we generally used 0.02, although any value within the range [0.015, 0.05] seemed to work fine. After the annealing process we performed a single step denoising to further enhance sample quality. and FID . We achieved Inception Score of 8.31 and FID of 31.7, comparable to modern GAN approaches. Scores for CelebA dataset are not reported here as they are not commonly reported and may depend on the specific pre-processing used. More samples and training images are provided in Appendix for visual inspection. We believe that visual assessment is still essential because of the possible issues with the Inception score . Indeed, we also found that the visually impressive samples were not necessarily the one achieving the highest Inception Score. Although overfitting is not a common concern for generative models, we still tested our model for overfitting. We found no indication for overfitting by comparing model samples with their nearest neighbors in the data set, see Figure C.1 in Appendix. Mode Coverage. We repeated with our model the 3 channel MNIST mode coverage experiment similar to the one in. An energy-based model was trained on 3-channel data where each channel is a random MNIST digit. Then 8000 samples were taken from the model and each channel was classified using a small MNIST classifier network. We obtained of the 966 modes, comparable to GAN approaches. Training was successful and our model assigned low energy to all the learned modes, but some modes were not accessed during sampling, likely due to the Langevin Dynamics failing to explore these modes. A better sampling technique such as or a Maximum Entropy Generator could improve this . Image Inpainting. Image impainting can be achieved with our model by clamping a part of the image to ground truth and performing the same annealed Langevin and Jump sampling procedure on the missing part of the image. Noise appropriate to the sampling temperature need to be added to the clamped inputs. The quality of inpainting of our model trained on CelebA and CIFAR-10 can be assessed in Figure 3. For CIFAR-10 inpainting we used the test set. Log likelihood estimation. For energy-based models, the log density can be obtained after estimating the partition function with Annealed Importance Sampling (AIS) or Reverse AIS . In our experiment on CIFAR-10 model, similar to reports in , there is still a substantial gap between AIS and Reverse AIS estimation, even after very substantial computational effort. In Table 1 We also report a density of 1.21 bits/dim on MNIST dataset, and we refer readers to for comparison to other models on this dataset. More details on this experiment is provided in the Appendix. Outlier Detection. and have reported intriguing behavior of high dimensional density models on out of distribution samples. Specifically, they showed that a lot of models assign higher likelihood to out of distribution samples than real data samples. We investigated whether our model behaves similarly. Our energy function is only trained outside the data manifold where samples are noisy, so the energy value at clean data points may not always be well behaved. Therefore, we added noise with magnitude σ 0 before measuring the energy value. We find that our network behaves similarly to previous likelihood models, it assigns lower energy, thus higher density, to some OOD samples. We show one example of this phenomenon in Appendix Figure F.1A. We also attempted to use the denoising performance, or the objective function to perform outlier detection. Intriguingly, the are similar as using the energy value. Denoising performance seems to correlate more with the variance of the original image than the content of the image. In this work we provided analyses and empirical for understanding the limitations of learning the structure of high-dimensional data with denoising score matching. We found that the objective function confines learning to a small set due to the measure concentration phenomenon in random vectors. Therefore, sampling the learned distribution outside the set where the gradient is learned does not produce good . One remedy to learn meaningful gradients in the entire space is to use samples during learning that are corrupted by different amounts of noise. applied this strategy very successfully. The central contribution of our paper is to investigate how to use a similar learning strategy in EBMs. Specifically, we proposed a novel EBM model, the Multiscale Denoising Score Matching (MDSM) model. The new model is capable of denoising, producing high-quality samples from random noise, and performing image inpainting. While also providing density information, our model learns an order of magnitude faster than models based on maximum likelihood. Our approach is conceptually similar to the idea of combining denoising autoencoder and annealing (; ;) though this idea was proposed in the context of pre-training neural networks for classification applications. Previous efforts of learning energy-based models with score matching (; were either computationally intensive or unable to produce high-quality samples comparable to those obtained by other generative models such as GANs. and trained energy-based model with the denoising score matching objective but the ing models cannot perform sample synthesis from random noise initialization. Recently, proposed the NCSN model, capable of high-quality sample synthesis. This model approximates the score of a family of distributions obtained by smoothing the data by kernels of different widths. The sampling in the NCSN model starts with sampling the distribution obtained with the coarsest kernel and successively switches to distributions obtained with finer kernels. Unlike NCSN, our method learns an energy-based model corresponding to p σ0 (x x x) for a fixed σ 0. This method improves score matching in high-dimensional space by matching the gradient of an energy function to the score of p σ0 (x x x) in a set that avoids measure concentration issue. All told, we offer a novel EBM model that achieves high-quality sample synthesis, which among other EBM approaches provides a new state-of-the art. Compared to the NCSN model, our model is more parsimonious than NCSN and can support single step denoising without prior knowledge of the noise magnitude. But our model performs sightly worse than the NCSN model, which could have several reasons. First, the derivation of Equation 6 requires an approximation to keep the training procedure tractable, which could reduce the performance. Second, the NCSNs output is a vector that, at least during optimization, does not always have to be the derivative of a scalar function. In contrast, in our model the network output is a scalar function. Thus it is possible that the NCSN model performs better because it explores a larger set of functions during optimization. In this section, we provide a formal discussion of the MDSM objective and suggest it as an improved score matching formulation in high-dimensional space. illustrated the connection between the model score −∇x x x E(x x x; θ) with the score of Parzen window density estimator ∇x x x log(p σ0 (x x x)). Specifically, the objective is Equation 1 which we restate here: Our key observation is: in high-dimensional space, due the concentration of measure, the expectation w.r.t. p σ0 (x x x) over weighs a thin shell at roughly distance √ dσ to the empirical distribution p(x). Though in theory this is not a problem, in practice this leads to that the score are only well matched on this shell. Based on this observation, we suggest to replace the expectation w.r.t. p σ0 (x x x) with a distribution p σ (x x x) that has the same support as p σ0 (x x x) but can avoid the measure concentration problem. We call this multiscale score matching and the objective is the following: Given that p M (x x x) and p σ0 (x x x) has the same support, it's clear that L M SM = 0 would be equivalent to L SM = 0. Due to the proof of the Theorem 2 in Hyvärinen, we have We follow the same procedure as in to prove this . Thus we have: The above analysis applies to any noise distribution, not limited to Gaussian. but L M DSM * has a reversed expectation form that is not easy to work with. To proceed further we study the case where q σ0 (x x x|x x x) is Gaussian and choose q M (x x x|x x x) as a Gaussian scale mixture and p M (x x x) = q M (x x x|x x x)p(x x x)dx. By Proposition 1 and Proposition 2, we have the following form to optimize: To minimize Equation (*), we can use the following importance sampling procedure : we can sample from the empirical distribution p(x x x), then sample the Gaussian scale mixture q M (x x x|x x x) and finally weight the sample by qσ 0 (x x x|x x x) q M (x x x|x x x). We expect the ratio to be close to 1 for the following reasons: Using Bayes rule, q σ0 (x x x|x x x) = p(x x x)qσ 0 (x x x|x x x) pσ 0 (x x x) we can see that q σ0 (x x x|x x x) only has support on discret data points x x x, same thing holds for q M (x x x|x x x). because inx x x is generated by adding Gaussian noise to real data sample, both estimators should give highly concentrated on the original sample point x x x. Therefore, in practice, we ignore the weighting factor and use Equation 6. Improving upon this approximation is left for future work. To compare with previous method, we trained energy-based model with denoising score matching using one noise level on MNIST, initialized the sampling with Gaussian noise of the same level, and sampled with Langevin dynamics at T = 1 for 1000 steps and perform one denoise jump to recover the model's best estimate of the clean sample, see Figure B.1. We used the same 12-layer ResNet as other MNIST experiments. Models were trained for 100000 steps before sampling. We demonstrate that the model does not simply memorize training examples by comparing model samples with their nearest neighbors in the training set. We use Fashion MNIST for this demonstration because overfitting can occur there easier than on more complicated datasets, see Figure C we used a 18-layer ResNet with 128 filters on the first layer. All network used the ELU activation function. We did not use any normalization in the ResBlocks and the filer number is doubled at each downsampling block. Details about the structure of our networks used can be found in our code release. All mentioned models can be trained on 2 GPUs within 2 days. Since the gradient of our energy model scales linearly with the noise, we expected our energy function to scale quadratically with noise magnitude. Therefore, we modified the standard energy-based network output layer to take a flexible quadratic form : where a i, c i, d i and b 1, b 2, b 3 are learnable parameters, and h i is the (flattened) output of last residual block. We found this modification to significantly improve performance compared to using a simple linear last layer. For CIFAR and CelebA we trained for 300k weight updates, saving a checkpoint every 5000 updates. We then took 1000 samples from each saved networks and used the network with the lowest FID score. For MNIST and fashion MNIST we simply trained for 100k updates and used the last checkpoint. During training we pad MNIST and Fashion MNIST to 32*32 for convenience and randomly flipped CelebA images. No other modification was performed. We only constrained the gradient of the energy function, the energy value itself could in principle be unbounded. However, we observed that they naturally stabilize so we did not explicitly regularize them. The annealing sampling schedule is optimized to improve sample quality for CIFAR-10 dataset, and consist of a total of 2700 steps. For other datasets the shape has less effect on sample quality, see Figure F.1 B for the shape of annealing schedule used. For the Log likelihood estimation we initialized reverse chain on test images, then sample 10000 intermediate distribution using 10 steps HMC updates each. Temperature schedule is roughly exponential shaped and the reference distribution is an isotropic Gaussian. The variance of estimation was generally less than 10% on the log scale. Due to the high variance of , and to avoid getting dominated by a single outlier, we report average of the log density instead of log of average density. We provide more inpainting examples and further demonstrate the mixing during sampling process in Figure E.1. We also provide more samples for readers to visually judge the quality of our sample generation in Figure E.2, E.3 and E.4. All samples are randomly selected. Energy values for CIFAR-10 train, CIFAR-10 test and SVHN datasets for a network trained on CIFAR-10 images. Note that the network does not over fit to the training set, but just like most deep likelihood model, it assigns lower energy to SVHN images than its own training data. B. Annealing schedule and a typical energy trace for a sample during Annealed Langevin Sampling. The energy of the sample is proportional to the temperature, indicating sampling is close to a quasi-static process.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

Learned energy based model with score matching

represent the short summary for finding the corresponding scientific article (long).

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the article for finding a single-sentence TLDR

Question: A restricted Boltzmann machine (RBM) learns a probabilistic distribution over its input samples and has numerous uses like dimensionality reduction, classification and generative modeling. Conventional RBMs accept vectorized data that dismisses potentially important structural information in the original tensor (multi-way) input. Matrix-variate and tensor-variate RBMs, named MvRBM and TvRBM, have been proposed but are all restrictive by construction. This work presents the matrix product operator RBM (MPORBM) that utilizes a tensor network generalization of Mv/TvRBM, preserves input formats in both the visible and hidden layers, and in higher expressive power. A novel training algorithm integrating contrastive divergence and an alternating optimization procedure is also developed. directly applicable to matrix and tensor data. The first RBM designed for tensor inputs is described in, where the visible layer is represented as a tensor but the hidden layer is still a vector. Furthermore, 24 the connection between the visible and hidden layers is described by a canonical polyadic (CP) tensor 25 decomposition, which constrains the model representation capability. Another RBM related 26 model that utilizes tensor input is the matrix-variate RBM (MvRBM). The visible and hidden 27 layers in an MvRBM are both matrices. Nonetheless, to limit the number of parameters, an MvRBM 28 models the connection between the visible and hidden layers through two separate matrices, which 29 restricts the ability of the model to capture correlations between different data modes. All these issues have motivated this work. Specifically, we propose a matrix product operator (MPO) DISPLAYFORM0 The "building blocks" of the MPO are the 4-way tensors DISPLAYFORM1, also called the MPO-cores. DISPLAYFORM2 DISPLAYFORM3 the summations in BID0 and are the key ingredients in being able to express generic weight tensors W. The storage complexity of an MPORBM with uniform ranks and dimensions is O(dIJR 2), which is layer tensors into a d-way tensor, which is then added elementwise with the corresponding bias tensor. The final step in the computation of the conditional probability is an elementwise application of the 56 logistic sigmoid function on the ing tensor. Let Θ = {B, C, W DISPLAYFORM0 with respect to the model parameter Θ. Similar to the standard RBM BID0, the expression of the 60 gradient of the log-likelihood is: DISPLAYFORM1 We mainly use the contrastive divergence (CD) procedure to train the MPORBM model. First, a Gibbs chain is initialized with one particular training sample V = X train, followed by K times Gibbs sampling which in the chain {( summing over all edges. The derivatives of the log-likelihood with respect to the bias tensors B, C are MPO-cores, which we call CD-SU henceforth, will be demonstrated through numerical experiments. DISPLAYFORM0 DISPLAYFORM1 In the first experiment, we demonstrate the superior data classification accuracy of MPORBM using Finally, we show that an MPORBM is good at generative modeling exemplified by image completion. We tested this generative task on the binarized MNIST dataset: one half of the image was provided to

Propose a general tensor-based RBM model which can compress the model greatly at the same keep a strong model expression capacity

Represent the short summary for finding the corresponding scientific article (long).

We propose an extension of conditional variational autoencoder that allows conditioning on an arbitrary subset of the features and sampling the remaining ones.


Question: represent the scientific article to find a short tldr:

Question: Autonomous driving is still considered as an “unsolved problem” given its inherent important variability and that many processes associated with its development like vehicle control and scenes recognition remain open issues. Despite reinforcement learning algorithms have achieved notable in games and some robotic manipulations, this technique has not been widely scaled up to the more challenging real world applications like autonomous driving. In this work, we propose a deep reinforcement learning (RL) algorithm embedding an actor critic architecture with multi-step returns to achieve a better robustness of the agent learning strategies when acting in complex and unstable environments. The experiment is conducted with Carla simulator offering a customizable and realistic urban driving conditions. The developed deep actor RL guided by a policy-evaluator critic distinctly surpasses the performance of a standard deep RL agent. An important approach for goal-oriented optimization is reinforcement learning (RL) inspired from behaviorist psychology BID25. The frame of RL is an agent learning through interaction with its environment driven by an impact (reward) signal. The environment return reinforces the agent to select new actions improving learning process, hence the name of reinforcement learning BID10. RL algorithms have achieved notable in many domains as games BID16 and advanced robotic manipulations BID13 beating human performance. However, standard RL strategies that randomly explore and learn faced problems lose efficiency and become computationally intractable when dealing with high-dimensional and complex environments BID26.Autonomous driving is one of the current highly challenging tasks that is still an "unsolved problem" more than one decade after the promising 2007 DARPA Urban Challenge BID4 ). The origin of its difficulty lies in the important variability inherent to the driving task (e.g. uncertainty of human behavior, diversity of driving styles, complexity of scene perception...).In this work, we propose to implement an advantage actor-critic approach with multi-step returns for autonomous driving. This type of RL has demonstrated good convergence performance and faster learning in several applications which make it among the preferred RL algorithms BID7. Actor-critic RL consolidates the robustness of the agent learning strategy by using a temporal difference (T D) update to control returns and guide exploration. The training and evaluation of the approach are conducted with the recent CARLA simulator BID6. Designed as a server-client system, where the server runs the simulation commands and renders the scene readings in return, CARLA is an interesting tool since physical autonomous urban driving generates major infrastructure costs and logistical difficulties. It particularly offers a realistic driving environment with challenging properties variability as weather conditions, illumination, and density of cars and pedestrians. The next sections review previous work on actor-critic RL and provide a detailed description of the proposed method. After presenting CARLA simulator and related application advantages, we evaluate our model using this environment and discuss experimental . Various types of RL algorithms have been introduced and are classified into three categories, actor, critic or actor-critic depending on whether they rely on a parameterized policy, a value function or a combination of both to predict actions BID12. In the actor-only methods, a gradient is generated to update the policy parameters in a direction of improvement BID28. Despite policy gradients offer tough convergence guarantees, they may suffer from high variance ing in slow learning BID2. On the other hand, critic-only methods built on value function approximation, use T D learning and show lower variance of estimated returns BID3. However, they lack reliable guarantee of converging and reaching the real optimum BID7.Actor-critic methods combine the advantages of the two previous ones by inducting a repetitive cycle of policy evaluation and improvement. BID1 is considered as the starting point that defined the basics of actor-critic algorithms commonly used in recent research. Since then, several algorithms have been developed with different directions of improvements. BID27, introduced the Fuzzy Actor-Critic Reinforcement Learning Network (FACRLN), which involves one neural network to approximate both the actor and the critic. Based on the same strategy, BID18 developed the Consolidated Actor-Critic Model (CACM). BID11 used for the first time a natural gradient BID0 for the policy updates in their actorcritic algorithm. BID23 presented the Deterministic Policy Gradient algorithm (DPG) that assign a learned value estimate to train a deterministic policy. Recently, BID17 proposed the Asynchronous Advantage Actor-Critic (A3C) algorithm where multiple agents operate in parallel allowing data decorrelation and learning experience diversity. Despite that several actor-critic methods have been developed, most of them were tested on standard RL benchmarks. The latter generally include basic tasks with low-level complexity comparatively to real world applications, like cart-pole balancing BID27 BID11, maze problems BID18, multi-armed bandit BID23, Atari games BID17 BID8 and OpenAI Gym tasks BID19. Our work contribution consists in extending actor-critic RL application to a very challenging task which is urban autonomous driving. The domain setting is particularly difficult to handle due to intricate and conflicting dynamics. Indeed, the driving agent must interact, in changing weather and lighting conditions and through a wide action space, with several actors that may behave unexpectedly, identify traffic rules and street lights, estimate appropriate speed and distance... Our approach, that will be detailed in the next section, incorporates an actor and a multi-step T D critic component to improve the stability of the RL method. The RL task considered in this work is a Markov Decision Process (MDP) T i defined according to the tuple (S, A, p, r, γ, ρ 0, H) where S is the set of states, A is the set of actions, p(s t+1 |s t, a t) is the state transition distribution predicting the probability to reach a state s t+1 in the next time step given current state and action, r is a reward function, γ is the discount factor, ρ 0 is the initial state distribution and H the horizon. Consider the sum of expected rewards (return) from a trajectory τ (0,H−1) = (s 0, a 0, ..., s H−1, a H−1, s H). A RL setting aims at learning a policy π of parameters θ (either deterministic or stochastic) that maps each state s to an optimal action a maximizing the return R of the trajectory. DISPLAYFORM0 Following the discounted return expressed above, we can define a state value function V (s): S → R and a state-action value function Q(s, a): A × S → R to measure, respectively, the current state and state-action returns estimated under policy π: DISPLAYFORM1 In value-based RL algorithms such as Q-learning, a value function is approximated to select the best action according to the maximum value attributed to each state and action pair. On the other hand, policy-based methods directly optimize a parameterized policy without using a value function. They use instead gradient descents like in the family of REINFORCE algorithms BID28 updating the policy parameters θ in the direction: DISPLAYFORM2 The main problem with policy based methods is that the score function R t uses the averaged rewards calculated at the end of a trajectory which may lead to the inclusion of "bad" actions and hence slow learning. The solution provided in actor-critic framework is to replace the reward function R t in the policy gradient (equation 4) with the action value function that will enable the agent to learn the long-term value of a state and therefore enhance its prediction decision: DISPLAYFORM3 Then train a critic to approximate this value function parameterized with ω and update the model accordingly. At this point, we can conclude that an efficient way to derive an optimal control of policies is to evaluate them using approximated value functions. Hence, building accurate value function estimators in better policy evaluation and faster learning. T D learning combining Monte Carlo method and dynamic programming BID25 has proved to be an effective way to calculate good approximations of value functions by allowing an efficient reuse of rewards during policy evaluation. It consists in taking an action according to the policy and bootstrapping the 1-step sampled return from the value function estimate ing in the below 1-step T D target: DISPLAYFORM4 Given the last return estimation, we obtain the 1-step T D update rule that allows the adjustment of the value function according to the T D error δ t with step size β: DISPLAYFORM5 At this level, the actor-critic algorithm still suffers from high variance. In order to reduce the variance of the policy gradient and stabilize learning, we can subtract a baseline function, e.g. the state value function, from the policy gradient. For that, we define the advantage function A(s t, a t) which calculates the improvement in predicting an action compared to the average V (s t): DISPLAYFORM6 An approximation of the advantage function is required since it involves two value functions Q(s t, a t) and V (s t). Therefore let's reformulate A(s t, a t) as the difference between the expected future reward and the actual reward that the agent receives from the environment BID9 ): DISPLAYFORM7 When used in the previous policy gradient (equation 5), this gives us the advantage of the actor policy gradient: DISPLAYFORM8 We can subsequently assume that T D error is a good candidate to estimate the advantage function. Accordingly, we deduce the final actor policy gradient: DISPLAYFORM9 Given the complex nature of the autonomous urban driving task, we will use a generalized version of T D learning by extending the bootstrapping over multiple time steps into the future. Algorithmically, we will define configurable multi-step returns within the T D target. Hence, T D error becomes: DISPLAYFORM10 Multi-step returns have been demonstrated to improve the performance of learning especially with the advent of deep RL BID17. Indeed, it allows the agent to gather more information on the environment before calculating the error in the critic estimates and updating the policy. So far, we have a good theoretical basis to launch our agent. The experiments carried out by the application of this approach in the Carla simulator will be presented in the next section. In this section we investigate the performance of an advantage actor-critic (A2C) algorithm embedding multi-step T D target updates on the challenging task of urban autonomous driving. The goal of our experimental evaluation is to demonstrate that the incorporation of a multi-step returns critic (MSRC) component in a deep RL framework consolidates the robustness of the agent by controlling and guiding its learning strategy. We expect a reduction of the actor gradient variance, an ascendant trend of episodic average returns and more generally a better performance comparatively to the case where the MSRC component is deactivated in the A2C algorithm. Environment. We conduct the experiments using CARLA simulator for autonomous driving which provides an interesting interface allowing our RL agent to control a vehicle and interact with a dynamic environment. Comparatively to existing platforms, Carla offers a customizable and quite realistic urban driving conditions with a set of advanced features for controlling the vehicle and gathering the environment feedback. It is designed as a server-client system where the server implemented in Unreal Engine 4 (UE4) 1 runs the simulation commands and returns the scene readings. The client implemented in Python sends the agent predicted actions mapped as driving commands and receives the ing simulation measures that will be interpreted as the agent rewards. Carla 3D environment consists of static objects as buildings, roads and vegetation and dynamic nonplayer characters, mainly pedestrians and vehicles. During training, we can episodically vary server settings as the traffic density (number of dynamic objects) and visual effects (weather and lightening conditions, sun position, cloudiness, precipitation...). Some examples of ing environments are illustrated in figure 1.Observation and action spaces. The agent interacts with the environment by generating actions and receiving observations over regular time steps. The action space selected for our experiments is built on the basis of three discrete driving instructions (steering, throttle, and brake) extended with some combinations in-between (turn left and accelerate/decelerate...). The observation space includes sensors outputs as color images produced by RGB cameras and derived depth and semantic segmentations. The second type of available observations consists in a range of measurements reporting the vehicle location (similarly to GPS) and speed, number of collisions, traffic rules and positioning of non-player dynamics characters. Rewards. A crucial role is played by rewards in building driving policies as they orient the agent predictions. In order to further optimal learning, the reward is shaped as a weighted sum of measurements extracted from the observations space described in the previous paragraph. The idea is to compute a difference between the current (step t) and the previous (step t − 1) measure of the selected observation then impact it positively or negatively on the aggregated reward. The positively weighted variables are distance traveled to target and speed in km/h. The negatively weighted variables are collisions damage (including collisions with vehicles, pedestrians and other), intersections with sidewalk and opposite lane. For example, the agent will get a reward if the distance to goal decreases and a penalty each time a collision or an intersection with the opposite lane is recorded. Experiment settings. The agent training follows a goal-directed navigation on straight roads from scratch. An episode is terminated when the target destination is reached or after a collision with a dynamic non-player character. The A2C networks are trained with 10 millions steps for 72 hours of simulated continuous driving. Motivated by the recent success achieved by deep RL in challenging domains BID17, we use convolutional neural networks (CNN) to approximate both the value function of the critic and the actor policy where the parameters are represented by the deep network weights. The CNN architectures consist of 4 convolutional layers, 3 max-pooling layers and one fully connected layer at the output. The discount factor is set as 0.9. We used 10-step rollouts, with initial learning rate set as 0, 0001. Learning rate is linearly decreased to zero over the course of training. While training the approach, a stochastic gradient descent is operated each 10 time steps and the ing policy model is stored only if its performance (accumulated rewards) exceeds the last retained model. The final stored model is then used in the test phase. Comparative evaluation. In the absence of various state-of-the-art works on the recent CARLA simulator, we choose to compare 2 versions of our algorithm: the original deep actor RL guided by the MSRC policy-evaluator versus a standard deep actor RL ing from the deactivation of the MSRC component in the original algorithm. In fact the few available state-of-the-art in CARLA environment BID6 BID14 report the percentage of successfully completed episodes. This type of quantitative evaluation doesn't meet our experiment objectives mentioned in the beginning of this section to evaluate and interpret the MSRC contribution in complex tasks like autonomous driving. Guided by the several works on RL strategies in different domains BID17, BID19, we selected episodic average and cumulative rewards metrics to evaluate our approach. FIG1 shows the generated reward in training phase. We use average episodic reward to describe the methods global performance and step reward to emphasize the predictions return variance. We can make few observations in this regard. In term of performance, our n-step A2C approach is dominant over almost all the 10000 training episodes confirming the efficiency of the RL strategy controlled by the MSRC. Furthermore, we noticed that regarding the best retained models, the A2C stored just few models in the 2000 first episodes, then this number drastically increased to 100 retained models in the remaining 8000 episodes. This means that our method early achieved the exploration phase and moved to exploitation from the training level of 2000 episodes. On the other hand, the standard deep RL totalized only 10 best models over the training phase reflecting the weak efficiency of a random strategy to solve a very complex and challenging problem like autonomous driving. A last visual interpretation that we can deduce from the step reward graph is that the variance of A2C predictions is significantly reduced relatively to the standard deep RL confirming the T D learning contribution in accomplishing a faster learning. Figure 3 recaps the testing phase evaluation following two different scenarios. First, the testing was conducted in the same environment and conditions as the training: Town 2 and Clear Noon weather (env1). From the episodic reward graph we can observe that our approach substantially outperforms the standard deep RL which means that training with multi-step returns critic leads to more efficient RL models. In the second scenario, both methods agents are tested in a different environment than training: Town 1 and in hard rainy conditions (env2). The n-step A2C is still more competitive than the standard deep RL showing superior generalization capabilities in the new unseen setting. Nevertheless, its performance has decreased in the second test scenario reflecting a certain fragility to changing environment. On the other side, the standard deep RL is still showing higher prediction return variance in the step reward graph confirming training phase . In this paper we addressed the limits of RL algorithms in solving high-dimensional and complex tasks. Combining both actor and critic methods advantages, the proposed approach implemented a continuous process of policy assessment and improvement using multi-step T D learning. Evaluated on the challenging problem of autonomous driving using CARLA simulator, our deep actor-critic algorithm demonstrated higher performance and faster learning capabilities than a standard deep RL. Furthermore, the showed a certain vulnerability of the approach when facing unseen testing conditions. Considering this paper as a preliminary attempt to scale up RL approaches to highdimensional real world applications like autonomous driving, we plan in future work to examine the performance of other RL methods such as deep Q-learning and Trust Region Policy Optimization BID22 on similar complex tasks. Furthermore, we propose to tackle the issue of non-stationary environments impact on RL methods robustness as a multi-task learning problem BID5. In such context, we will explore recently applied concepts and methodologies such as novel adaptive dynamic programming (ADP) approaches, context-aware and meta-learning strategies. The latter are currently attracting a keen research interest and particularly achieving promising advances in designing generalizable and fast adapting RL algorithms BID21 BID20. Subsequently, we will be able to increase driving tasks complexity and operate conclusive comparisons with the few available state-of-the-art experiments on CARLA simulator.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An actor-critic reinforcement learning approach with multi-step returns applied to autonomous driving with Carla simulator.

Represent the short summary for finding the corresponding scientific article (long)

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.