Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition \--- given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method. Word embeddings have become one of the most popular techniques in natural language processing. A word embedding maps each word in the vocabulary to a low dimensional vector. Several algorithms (e.g., ;) can produce word embedding vectors whose distances or inner-products capture semantic relationships between words. The vector representations are useful for solving many NLP tasks, such as analogy tasks or serving as features for supervised learning problems .While word embeddings are good at capturing the semantic information of a single word, a key challenge is the problem of composition: how to combine the embeddings of two co-occurring, syntactically related words to an embedding of the entire phrase. In practice composition is often done by simply adding the embeddings of the two words, but this may not be appropriate when the combined meaning of the two words differ significantly from the meaning of individual words (e.g., "complex number" should not just be "complex"+"number").In this paper, we try to learn a model for word embeddings that incorporates syntactic information and naturally leads to better compositions for syntactically related word pairs. Our model is motivated by the principled approach for understanding word embeddings initiated by , and models for composition similar to. gave a generative model (RAND-WALK) for word embeddings, and showed several previous algorithms can be interpreted as finding the hidden parameters of this model. However, the RAND-WALK model does not treat syntactically related word-pairs differently from other word pairs. We give a generative model called syntactic RAND-WALK (see Section 3) that is capable of capturing specific syntactic relations (e.g., adjective-noun or verb-object pairs). Taking adjective-noun pairs as an example, previous works (; ;) have tried to model the adjective as a linear operator (a matrix) that can act on the embedding of the noun. However, this would require learning a d × d matrix for each adjective while the normal embedding only has dimension d. In our model, we use a core tensor T ∈ R d×d×d to capture the relations between a pair of words and its context. In particular, using the tensor T and the word embedding for the adjective, it is possible to define a matrix for the adjective that can be used as an operator on the embedding of the noun. Therefore our model allows the same interpretations as many previous models while having much fewer parameters to train. One salient feature of our model is that it makes good use of high order statistics. Standard word embeddings are based on the observation that the semantic information of a word can be captured by words that appear close to it. Hence most algorithms use pairwise co-occurrence between words to learn the embeddings. However, for the composition problem, the phrase of interest already has two words, so it would be natural to consider co-occurrences between at least three words (the two words in the phrase and their neighbors).Based on the model, we can prove an elegant relationship between high order co-occurrences of words and the model parameters. In particular, we show that if we measure the Pointwise Mutual Information (PMI) between three words, and form an n × n × n tensor that is indexed by three words a, b, w, then the tensor has a Tucker decomposition that exactly matches our core tensor T and the word embeddings (see Section 2, Theorem 1, and Corollary 1). This suggests a natural way of learning our model using a tensor decomposition algorithm. Our model also allows us to approach the composition problem with more theoretical insights. Based on our model, if words a, b have the particular syntactic relationships we are modeling, their composition will be a vector v a + v b + T (v a, v b, ·). Here v a, v b are the embeddings for word a and b, and the tensor gives an additional correction term. By choosing different core tensors it is possible to recover many previous composition methods. We discuss this further in Section 3.Finally, we train our new model on a large corpus and give experimental evaluations. In the experiments, we show that the model learned satisfies the new assumptions that we need. We also give both qualitative and quantitative for the new embeddings. Our embeddings and the novel composition method can capture the specific meaning of adjective-noun phrases in a way that is impossible by simply "adding" the meaning of the individual words. Quantitative experiment also shows that our composition vector are better correlated with humans on a phrase similarity task. Syntax and word embeddings Many well-known word embedding methods (e.g., ;) don't explicitly utilize or model syntactic structure within text. find that such syntax-blind word embeddings fail to capture syntactic information above and beyond what a statistical parser can obtain, suggesting that more work is required to build syntax into word embeddings. Several syntax-aware embedding algorithms have been proposed to address this. Levy & Goldberg (2014a) propose a syntax-oriented variant of the well-known skip-gram algorithm of , using contexts generated from syntactic dependency-based contexts obtained with a parser. build syntax-awareness into a neural network model for word embeddings by indroducing a negative set of samples in which the order of the context words is shuffled, in hopes that the syntactic elements which are sensitive to word order will be captured. Word embedding composition Several works have addressed the problem of composition for word embeddings. On the theoretical side, give a theoretical justification for additive embedding composition in word models that satisfy certain assumptions, such as the skipgram model, but these assumptions don't address syntax explicitly. present a mathematical framework for reasoning about syntax-aware word embedding composition that motivated our syntactic RAND-WALK model. Our new contribution is a concrete and practical learning algorithm with theoretical guarantees. Mitchell & Lapata (2008; 2010) explore various composition methods that involve both additive and multiplicative interactions between the component embeddings, but some of these are limited by the need to learn additional parameters post-hoc in a supervised fashion. get around this drawback by first training word embeddings for each word and also for tokenized adjective-noun pairs. Then, the composition model is trained by using the constituent adjective and noun embeddings as input and the adjective-noun token embedding as the predictive target. treat adjectives as matrices and nouns as vectors, so that the composition of an adjective and noun is just matrix-vector multiplication. The matrices and vectors are learned through an extension of the skip-gram model with negative sampling. In contrast to these approaches, our model gives rise to a syntax-aware composition function, which can be learned along with the word embeddings in an unsupervised fashion, and which generalizes many previous composition methods (see Section 3.3 for more discussion).Tensor factorization for word embeddings As Levy & Goldberg (2014b) and point out, some popular word embedding methods are closely connected matrix factorization problems involving pointwise mutual information (PMI) and word-word co-occurrences. It is natural to consider generalizing this basic approach to tensor decomposition. demonstrate this technique by performing a CP decomposition on triple word co-occurrence counts. explore this idea further by defining a third-order generalization of PMI, and then performing a symmetric CP decomposition on the ing tensor. In contrast to these recent works, our approach arives naturally at the more general Tucker decomposition due to the syntactic structure in our model. Our model also suggests a different (yet still common) definition of third-order PMI. Notation For a vector v, we use v to denote its Euclidean norm. For vectors u, v we use u, v to denote their inner-product. For a matrix M, we use M to denote its spectral norm, DISPLAYFORM0 to denote its Frobenius norm, and M i,: to denote it's i-th row. In this paper, we will also often deal with 3rd order tensors, which are just three-way indexed arrays. We use ⊗ to denote the tensor product: DISPLAYFORM1 Tensor basics Just as matrices are often viewed as bilinear functions, third order tensors can be interpreted as trilinear functions over three vectors. Concretely, let T be a d × d × d tensor, and let x, y, z ∈ R d. We define the scalar T (x, y, z) ∈ R as follows DISPLAYFORM2 This operation is linear in x, y and z. Analogous to applying a matrix M to a vector v (with the vector M v), we can also apply a tensor T to one or two vectors, ing in a matrix and a vector, respectively: DISPLAYFORM3 We will make use of the simple facts that z, T (x, y, ·) = T (x, y, z) and [T (x, ·, ·)] y = T (x, y, ·).Tensor decompositions Unlike matrices, there are several different definitions for the rank of a tensor. In this paper we mostly use the notion of Tucker rank. A tensor T ∈ R n×n×n has Tucker rank d, if there exists a core tensor S ∈ R d×d×d and matrices A, B, C ∈ R n×d such that DISPLAYFORM4 The equation above is also called a Tucker decomposition of the tensor T. The Tucker decomposition for a tensor can be computed efficiently. When the core tensor S is restricted to a diagonal tensor (only nonzero at entries S i,i,i), the decomposition is called a CP decomposition; which can also be written as DISPLAYFORM5. In this case, the tensor T is the sum of d rank-1 tensors (A i,: ⊗ B i,: ⊗ C i,:). However, unlike matrix factorizations and the Tucker decomposition, the CP decomposition of a tensor is hard to compute in the general case (Håstad, 1990;). Later in Section 4 we will also see why our model for syntactic word embeddings naturally leads to a Tucker decomposition. In this section, we introduce our syntactic RAND-WALK model and present formulas for inference in the model. We also derive a novel composition technique that emerges from the model. We first briefly review the RAND-WALK model . In this model, a corpus of text is considered as a sequence of random variables w 1, w 2, w 3,..., where w t takes values in a vocabulary V of n words. Each word w ∈ V has a word embedding v w ∈ R d. The prior for the word embeddings is v w = s ·v, where s is a positive bounded scalar random variable with constant expectation τ and upper bound κ, andv ∼ N (0, I).The distribution of each w t is determined in part by a random walk {c t ∈ R d | t = 1, 2, 3 . . .}, where c t -called a discourse vector -represents the topic of the text at position t. This random walk is slow-moving in the sense that c t+1 − c t is small, but mixes quickly to a stationary distribution that is uniform on the unit sphere, which we denote by C.Let C denote the sequence of discourse vectors, and let V denote the set of word embeddings. Given these latent variables, the model specifies the following conditional probability distribution: DISPLAYFORM0 The graphical model depiction of RAND-WALK is shown in FIG0. One limitation of RAND-WALK is that it can't deal with syntactic relationships between words. Observe that conditioned on c t and V, w t is independent of the other words in the text. However, in natural language, words can exhibit more complex dependencies, e.g. adjective-noun pairs, subject-verb-object triples, and other syntactic or grammatical structures. In our syntactic RAND-WALK model, we start to address this issue by introducing direct pairwise word dependencies in the model. When there is a direct dependence between two words, we call the two words a syntactic word pair. In RAND-WALK, the interaction between a word embedding v and a discourse vector c is mediated by their inner product v, c. When modeling a syntactic word pair, we need to mediate the interaction between three quantities, namely a discourse vector c and the word embeddings v and v of the two relevant words. A natural generalization is to use a trilinear form defined by a tensor T, i.e. DISPLAYFORM0 Here, T ∈ R d×d×d is also a latent random variable, which we call the composition tensor. We model a syntactic word pair as a single semantic unit within the text (e.g. in the case of adjectivenoun phrases). We realize this choice by allowing each discourse vector c t to generate a pair of words w t, w t with some small probability p syn. To generate a syntactic word pair w t, w t, we first generate a root word w t conditioned on c t with probability proportional to exp(c t, w t), and then we draw w t from a conditional distribution defined as follows: DISPLAYFORM1 Here exp(c t, v b) would be proportional to the probability of generating word b in the original RAND-WALK model, without considering the syntactic relationship. The additional term T (v a, v b, c t) can be viewed as an adjustment based on the syntactic relationship. We call this extended model Syntactic RAND-WALK. FIG0 gives the graphical model depiction for a syntactic word pair, and we summarize the model below. Definition 1 (Syntactic RAND-WALK model). The model consists of the following:1. Each word w in vocabulary has a corresponding embedding v w ∼ s ·v w, where s ∈ R ≥0 is bounded by κ and DISPLAYFORM2 2. The sequence of discourse vectors c 1,..., c t are generated by a random walk on the unit sphere, c t − c t+1 ≤ w / √ d and the stationary distribution is uniform.3. For each c t, with probability 1−p syn, it generates one word w t with probability proportional to exp(c t, v wt).4. For each c t, with probability p syn, it generates a syntactic pair w t, w t with probability proportional to exp(c t, v wt) and exp(c t, DISPLAYFORM3 We now calculate the marginal probabilities of observing pairs and triples of words under the syntactic RAND-WALK model. We will show that these marginal probabilities are closely related to the model parameters (word embeddings and the composition tensor). All proofs in this section are deferred to supplementary material. Throughout this section, we consider two adjacent context vectors c t and c t+1, and condition on the event that c t generated a single word and c t+1 generated a syntactic pair 1. The main bottleneck in computing the marginal probabilities is that the conditional probailities specified in equations FORMULA6 and are not normalized. Indeed, for these equations to be exact, we would need to divide by the appropriate partition functions, namely Z ct:= w∈V exp(v w, c t) for the former and Z ct,a:= w∈V exp(c t, v w + T (v a, v w, c t)) for the latter. Fortunately, we show that under mild assumptions these quantities are highly concentrated. To do that we need to control the norm of the composition tensor. Definition 2. The composition tensor T is (K,)-bounded, if for any word embedding v a, v b, we have DISPLAYFORM0 To make sure exp(c t, v w +T (v a, v w, c t)) are within reasonable ranges, the value K in this definition should be interpreted as an absolute constant (like 5, similar to previous constants κ and τ). Intuitively these conditions make sure that the effect of the tensor cannot be too large, while still making sure the tensor component T (v a, v b, c) can be comparable (or even larger than) v b, c. We have not tried to optimize the log factors in the constraint for T (v a, ·, ·) + I 2.Note that if the tensor component T (v a, ·, ·) has constant singular values (hence comparable to I), we know these conditions will be satisfied with K = O and = O(DISPLAYFORM1). Later in Section 5 we verify that the tensors we learned indeed satisfy this condition. Now we are ready to state the concentration of partition functions:Lemma 1 (Concentration of partition functions). For the syntactic RAND-WALK model, there exists a constant Z such that Pr DISPLAYFORM2 Furthermore, if the tensor T is (K,)-bounded, then for any fixed word a ∈ V, there exists a constant Z a such that Pr DISPLAYFORM3 Using this lemma, we can obtain simple expressions for co-occurrence probabilities. In particular, for any fixed w, a, b ∈ V, we adopt the following notation: DISPLAYFORM4 Here in particular we use [a, b] to highlight the fact that a and b form a syntactic pair. Note p(w, a) is the same as the co-occurrence probability of words w and a if both of them are the only word generated by the discourse vector. Later we will also use p(w, b) to denote Pr[DISPLAYFORM5 We also require two additional properties of the word embeddings, namely that they are norm-bounded above by some constant times DISPLAYFORM6 and that all partition functions are bounded below by a positive constant. Both of these properties hold with high probability over the word embeddings provided n d log d and d log n, as shown in the following lemma: Lemma 2. Assume that the composition tensor T is (K,)-bounded, where K is a constant. With probability at least 1 − δ 1 − δ 2 over the word vectors, where DISPLAYFORM7, there exist positive absolute constants γ and β such that v i ≤ κγ for each i ∈ V and Z c ≥ β and Z c,a ≥ β for any unit vector c ∈ R d and any word a ∈ V.We can now state the main . Theorem 1. Suppose that the events referred to in Lemma 1 hold. Then DISPLAYFORM8 DISPLAYFORM9 where is from the (K,)-boundedness of T and w is from Definition 1. Our model suggests that the latent discourse vectors contain the meaning of the text at each location. It is therefore reasonable to view the discourse vector c corresponding to a syntactic word pair (a, b) as a suitable representation for the phrase as a whole. The posterior distribution of c given (a, b) satisfies DISPLAYFORM0 Since Pr[c t = c] is constant, and since Z c and Z c,a concentrate on values that don't depend on c, the MAP estimate of c given [a, b], which we denote byĉ, satisfieŝ DISPLAYFORM1 Hence, we arrive at our basic tensor composition: for a syntactic word pair (a, b), the composite embedding for the phrase is DISPLAYFORM2 Note that our composition involves the traditional additive composition DISPLAYFORM3 e. the composition tensor allows us to compactly associate a matrix with each word in the same vein as. Depending on the actual value of T, the term T (v a, v b, ·) can also recover any manner of linear or multiplicative interactions between v a and v b, such as those proposed in. In this section we discuss how to learn the parameters of the syntactic RAND-WALK model. Theorem 1 provides key insights into the learning problem, since it relates joint probabilities between words (which can be estimated via co-occurrence counts) to the word embeddings and composition tensor. By examining these equations, we can derive a particularly simple formula that captures these relationships. To state this equation, we define the PMI for 3 words as DISPLAYFORM0 We note that this is just one possible generalization of pointwise mutual information (PMI) to several random variables, but in the context of our model, it is a very natural definition as all the partition numbers will be canceled out. Indeed, as an immediate corollary of Theorem 1, we have Corollary 1. Suppose that the events referred to in Lemma 1 hold. Then for p same as Theorem 1 DISPLAYFORM1 That is, if we consider P M I3(a, b, w) as a n × n × n tensor, Equation equation 8 is exactly a Tucker decomposition of this tensor of Tucker rank d. Therefore, all the parameters of the syntactic RAND-WALK model can be obtained by finding the Tucker decomposition of the PMI3 tensor. This equation also provides a theoretical motivation for using third-order pointwise mutual information in learning word embeddings. We now discuss concrete details about our implementation of the learning algorithm. Corpus. We train our model using a February 2018 dump of the English Wikipedia. The text is pre-processed to remove non-textual elements, stopwords, and rare words (words that appear less than 1000 within the corpus), ing in a vocabulary of size 68,279. We generate a matrix of word-word co-occurrence counts using a window size of 5. To generate the tensors of adjective-noun-word and verb-object-word co-occurrence counts, we first run the Stanford Dependency Parser on the corpus in order to identify all adjective-noun and verb-object word pairs, and then use context windows that don't cross sentence boundaries to populate the triple co-occurrence counts. Training. We first train the word embeddings according to the RAND-WALK model, following. Using the learned word embeddings, we next train the composition tensor T via the following optimization problem DISPLAYFORM0 where X (a,b),w denotes the number of co-occurrences of word w with the syntactic word pair (a, b) (a denotes the noun/object) and f (x) = min(x, 100). This objective function isn't precisely targeting the Tucker decomposition of the PMI3 tensor, but it is analogous to the training criterion used in , and can be viewed as a negative log-likelihood for the model. To reduce the number of parameters, we constrain T to have CP rank 1000. We also trained the embeddings and tensor jointly, but found that this approach yields very similar . In all cases, we utilize the Tensorflow framework BID0 with the Adam optimizer (using default parameters), and train for 1-5 epochs. In this section, we verify and evaluate our model empirically on select qualitative and quantitative tasks. In all of our experiments, we focus solely on syntactic word pairs formed by adjective-noun phrases, where the noun is considered the root word. empirically verify the model assumptions of RAND-WALK, and since we trained our embeddings in the same way, we don't repeat their verifications here. Instead, we verify two key properties of syntactic RAND-WALK. We check the assumptions that the tensor T is (K,)-bounded. Ranging over all adjective-noun pairs in the corpus, we find that DISPLAYFORM0 2 has mean 0.052 and maximum 0.248, DISPLAYFORM1 F has mean 1.61 and maximum 3.23, and DISPLAYFORM2 has mean 0.016 and maximum 0.25. Each of these three quantities has a well-bounded mean, but T (v a, ·, ·) + I 2 has some larger outliers. If we ignore the log factors (which are likely due to artifacts in the proof) in Definition 2, the tensor is (K,) bounded for K = 4 and = 0.25. In addition to Definition 2, we also directly check its implications: our model predicts that the partition functions Z c,a concentrate around their means. To check this, given a noun a, we draw 1000 random vectors c from the unit sphere, and plot the histogram of Z c,a.Results for a few randomly selected words a are given in FIG2. All partition functions that we inspected exhibited good concentration. We test the performance of our new composition for adjective-noun and verb-object pairs by looking for the words with closest embedding to the composed vector. For a phrase (a, b), we compute c = v a + v b + T (v a, v b, ·), and then retrieve the words w whose embeddings v w have the largest cosine similarity to c. We compare our to the additive composition method. TAB0 show for three adjective-noun and verb-object phrases. In each case, the tensor composition is able to retrieve some words that are more specifically related to the phrase. However, the tensor composition also sometimes retrieves words that seem unrelated to either word in the phrase. We conjecture that this might be due to the sparseness of co-occurrence of three words. We also observed cases where the tensor composition method was about on par with or inferior to the additive composition method for retrieving relevant words, particularly in the case of low-frequency phrases. More can be found in supplementary material. Published as a conference paper at ICLR 2019 We also test our tensor composition method on a adjective-noun phrase similarity task using the dataset introduced by. The data consists of 108 pairs each of adjective-noun and verb-object phrases that have been given similarity ratings by a group of 54 humans. The task is to use the word embeddings to produce similarity scores that correlate well with the human scores; we use both the Spearman rank correlation and the Pearson correlation as evaluation metrics for this task. We note that the human similarity judgments are somewhat noisy; intersubject agreement for the task is 0.52 as reported in.Given a phrase (a, b) with embeddings v a, v b, respectively, we found that the tensor composition FORMULA6, we split the data into a development set of 18 humans and a test set of the remaining 36 humans. We use the development set to select the optimal scalar weight for the weighted tensor composition, and using this fixed parameter, we report the using the test set. We repeat this three times, rotating over folds of 18 subjects, and report the average . DISPLAYFORM0 DISPLAYFORM1 yields worse performance than the simple additive composition v a + v b. For this reason, we consider a weighted tensor composition vAs a baseline, we also report the average using just the additive composition, as well as a weighted additive composition βv a + v b, where β ≥ 0. We select β using the development set ("weighted1") and the test set ("weighted2"). We allow weighted2 to cheat in this way because it provides an upper bound on the best possible weighted additive composition. Additionally, we compare our method to the smoothed inverse frequency ("sif") weighting method that has been demonstrated to be near state-of-the-art for sentence embedding tasks . We also test embeddings of the form p + γω a ω b T (v a, v b, ·) ("sif+tensor"), where p is the sif embedding for (a, b), ω a and ω b are the smoothed inverse frequency weights used in the sif embeddings, and γ is a positive weight selected using the development set. The motivation for this hybrid embedding is to evaluate the extent to which the sif embedding and tensor component can independently improve performance on this task. We perform these same experiments using two other standard sets of pre-computed word embeddings, namely GloVe 3 and carefully optimized cbow vectors 4 . We re-trained the composition tensor using the same corpus and technique as before, but substituting these pre-computed embeddings in place of the RAND-WALK (rw) embeddings. However, a bit of care must be taken here, since our syntactic RAND-WALK model constrains the norm of the word embeddings to be related to the frequency of the words, whereas this is not the case with the pre-computed embeddings. To deal with this, we rescaled the pre-computed embeddings sets to have the same norms as their counterparts in the rw embeddings, and then trained the composition tensor using these rescaled embeddings. At test time, we use the original embeddings to compute the additive components of our compositions, but use the rescaled versions when computing the tensor components. The for adjective-noun phrases are given in Tables 3. We observe that the tensor composition outperforms the additive compositions on all embedding sets apart from the Spearman correlation on the cbow vectors, where the weighted additive 2 method has a slight edge. The sif embeddings outperform the additive and tensor methods, but combining the sif embeddings and the tensor components yields the best performance across the board, suggesting that the composition tensor captures additional information beyond the individual word embeddings that is useful for this task. There was high consistency across the folds for the optimal weight parameter α, with α = 0.4 for the rw embeddings, α =.2,.3 for the glove embeddings, and α =.3 for the cbow embeddings. For the sif+tensor embeddings, γ was typically in the range [.1, .2].The for verb-object phrases are given in Table 4. Predicting phrase similarity appears to be harder in this case. Notably, the sif embeddings perform worse than unweighted vector addition. As before, we can improve the sif embeddings by adding in the tensor component. The tensor composition method achieves the best for the glove and cbow vectors, but weighted addition works best for the randwalk vectors. Overall, these demonstrate that the composition tensor can improve the quality of the phrase embeddings in many cases, and the improvements are at least somewhat orthogonal to improvements In this section we present additional qualitiative demonstrating the use of the composition tensor for the retrieval of words related to adjective-noun and verb-object phrases. In TAB3, we show for the phrases "giving birth", "solve problem", and "changing name". These phrases are all among the top 500 most frequent verb-object phrases appearing in the training corpus. In these examples, the tensor-based phrase embeddings retrieve words that are generally markedly more related to the phrase at hand, and there are no strange false positives. These examples demonstrate how a verb-object phrase can encompass an action that isn't implied simply by the object or verb alone. The additive composition doesn't capture this action as well as the tensor composition. Moving on to adjective-noun phrases, in TAB4, we show for the phrases "United States", "Soviet Union", and "European Union". These phrases, which all occur with comparatively high frequency in the corpus, were identified as adjective-noun phrases by the tagger, but they function more as compound proper nouns. In each case, the additive composition retrieves reasonably relevant words, while the tensor composition is more of a mixed bag. In the case of "European Union", the tensor composition does retrieve the highly relevant words eec (European Economic Community) and eea (European Economic Area), which the additive composition misses, but the tensor composition also produces several false positives. It seems that for these types of phrases, the additive composition is sufficient to capture the meaning. In TAB5, we fix the noun "taste" and vary the modifying adjective to highlight different senses of the noun. In the case of "expensive taste", both compositions retrieve words that seem to be either related to "expensive" or "taste", but there don't seem to be words that are intrinsically related to the phrase as a whole (with the exception, perhaps, of "luxurious", which the tensor composition retrieves). In the case of "awful taste", both compositions retrieve fairly similar words, which mostly relate to the physical sense of taste (rather than the more abstract sense of the word). For the phrase "refined taste", the additive composition fails to capture the sense of the phrase and retrieves many words related to food taste (which are irrelevant in this context), whereas the tensor composition retrieves more relevant words. In TAB6, we fix the noun "friend" and vary the modifying adjective, but in all three cases, the adjective-noun phrase has basically the same meaning. In the case of "close friend" and "dear friend", both compositions retrieve fairly relevant and similar words. In the case of "best friend", both compositions retrieve false positives: the additive composition seems to find words related to movie awards, while the tensor composition finds unintuitive false positives. We note that in all three phrases, the tensor composition consistently retrieves the words "confidante", "confided" or "confides", "coworker", and "protoge", all of which are fairly relevant. We test the effect of using the composition tensor for a sentiment analysis task. We use the movie review dataset of Pang and Lee as well as the Large Movie Review dataset , which consist of 2,000 movie reviews and 50,000 movie reviews, respectively. For a fixed review, we identify each adjective-noun pair (a, b) and compute T (v a, v b, ·). We add these compositions together with the word embeddings for all of the words in the review, and then normalize the ing sum. This vector is used as the input to a regularized logistic regression classifier, which we train using scikit-learn with the default parameters. We also consider a baseline method where we simply add together all of the word embeddings in the movie review, and then normalize the sum. We evaluate the test accuracy of each method using TAB7. Although the tensor method seems to have a slight edge over the baseline, the differences are not significant. In this section we will prove the main Theorem 1, which establishes the connection between the model parameters and the correlations of pairs/triples of words. As we explained in Section 3, a crucial step is to analyze the partition function of the model and show that the partition functions are concentrated. We will do that in Section B.1. We then prove the main theorem in Section B.2. More details and some technical lemmas are deferred to Section B.3 In this section we will prove concentrations of partition functions (Lemma 1). Recall that we need the tensor to be K-bounded (where K is a constant) for this to work. DISPLAYFORM0 Note that K here should be considered as an absolute constant (like 5, in fact in Section 5 we show K is less than 4). We first restate Lemma 1 here: Lemma 3 (Lemma 1 restated). For the syntactic RAND-WALK model, there exists a constant Z such that Pr DISPLAYFORM1 Furthermore, if the tensor T is (K,)-bounded, then for any fixed word a ∈ V, there exists a constant Z a such that Pr DISPLAYFORM2 In fact, the first part of this Lemma is exactly Lemma 2.1 in. Therefore we will focus on the proof of the second part. For the second part, we know the probability of choosing a word b is proportional to DISPLAYFORM3 If the probability of choosing word w is proportional to exp(r, v w) for some vector r (think of r = T (v a, ·, c)+c), then in expectation the partition function should be equal to nE v∼D V [exp( r, v)] (here D V is the distribution of word embedding). When the number of words is large enough, we hope that with high probability the partition function is close to its expectation. Since the Gaussian distribution is spherical, we also know that the expected partition function nE v∼D V [exp( r, v)] should only depend on the norm of r. Therefore as long as we can prove the norm of r = T (v a, ·, c)+c remain similar for most c, we will be able to prove the desired in the lemma. We will first show the norm of r = T (v a, ·, c) + c is concentrated if the tensor T is (K,)-bounded. Throughout all subsequent proofs, we assume that < 1 and d ≥ log 2 n/ 2.Lemma 4. Let v a be a fixed word vector, and let c be a random discourse vector. If T is (K,)-bounded with d ≥ log 2 n/ 2, we have DISPLAYFORM4 where 0 ≤ L ≤ K is a constant that depends on v a, and δ = exp(−Ω(log 2 n)).Proof. Since c is a uniform random vector on the unit sphere, we can represent c as c = z/ z, where z ∼ N (0, I) is a standard spherical Gaussian vector. For ease of notation, let M = T (v a, ·, ·)+I, and write the singular value decomposition of M as M = U ΣV T. Note that Σ = diag(λ 1, . . ., λ d) and U and V are orthogonal matrices, so that in particular, the random variable y = V T z has the same distribution as z, i.e. its entries are i.i.d. standard normal random variables. Further, U x 2 = x 2 for any vector x, since U is orthogonal. Hence, we have DISPLAYFORM5 Since both the numerator and denominator of this quantity are generalized χ 2 random variables, we can apply Lemma 7 to get tail bounds on both. Observe that by assumption, we have λ 2 i ≤ Kd 2 / log 2 n for all i, and DISPLAYFORM6 We will apply Lemma 7 to prove concentration bounds for A, in this case we have DISPLAYFORM7 Under our assumptions, we know λ 2 max ≤ Kd 2 / log 2 n and DISPLAYFORM8 Similarly, we can apply Lemma 7 to B (in fact we can apply simpler concentration bounds for standard χ 2 distribution), and we get DISPLAYFORM9 If we take x = 1 16 log 2 n, we know 2 DISPLAYFORM10 When both events happen we know considered as a constant). This finishes the proof. DISPLAYFORM11 Using this lemma, we will show that the expected condition number nE v∼D V [exp( r, v)] (where r = T (v a, ·, c) + c) is concentrated Lemma 5. Let v a be a fixed word vector, and let c be a random discourse vector. If T is (K,)-bounded, there exists Z a such that we have DISPLAYFORM12 where Z a = Θ(n) depends on v a, and δ = exp(−Ω(log 2 n)).Proof. We know v = s ·v wherev ∼ N (0, I) and s is a (random) scaling. Let r = T (v a, ·, c) + c. Conditioned on s we know r, v is equivalent to a Gaussian random variable with standard deviation σ = r s. For this random variable we know DISPLAYFORM13. In particular, this implies g(x + γ) ≤ exp(κ 2 γ/2)g(x) (for small γ).By Lemma 4, we know with probability at least 1 − Ω(log 2 n), r 2 ∈ L ± O. Therefore, when this holds, we have DISPLAYFORM14 The multiplicative factor on the RHS is bounded by 1 + O when is small enough (and κ is a constant). This finishes the proof. Now we know the expected partition function is concentrated (for almost all discourse vectors c), it remains to show when we have finitely many words the partition function is concentrated around its expectation. This was already proved in Arora et al. FORMULA6, we use their lemma below: Lemma 6. For any fixed vector r (whose norm is bounded by a constant), with probability at least 1 − exp(−Ω(log 2 n)) over the choices of the words, we have DISPLAYFORM15 DISPLAYFORM16 This is essentially Lemma 2.1 in (see Equation A.32). The version we stated is a bit different because we allow r to have an arbitrary constant norm (while in their proof vector r is the discourse vector c and has norm 1). This is a trivial corollary as we can move the norm of r into the distribution of the scaling factor s for the word embedding. Finally we are ready to prove Lemma 1.Proof of Lemma 1. The first part is exactly Lemma 2.1 in.For the second part, note that the partition function DISPLAYFORM17 We will use E[Z c,a] to denote its expectation over the randomness of the word embedding {v i}. By Lemma 5, we know for at least 1 − exp(−Ω(log 2 n)) fraction of discourse vectors c, the expected partition function is concentrated (E[Z c,a] ∈ (1 ± O)Z a ). Let S denote the set of c such that Lemma 5 holds. Now by Lemma 6 we know for any x ∈ S, with probability at least 1 DISPLAYFORM18 Therefore we know if we consider both c and the embedding as random variables, Pr[Z c,a ∈ (1 ± O( + z))Z a ] ≥ 1 − δ where δ = exp(−Ω(log 2 n)). Let S be the set of word embedding such that there is at least DISPLAYFORM19 That is, with probability at least 1 − √ δ (over the word embeddings), there is at least 1 − √ δ fraction of c such that Z c,a ∈ (1 ± O( + z))Z a. In this section we prove Theorem 1 and Corollary 1. The proof is very similar to the proof of Theorem 2.2 in. We use several lemmas in that proof, and these lemmas are deferred to Section B.3.Proof of Theorem 1. Throughout this proof we consider two adjacent discourse vectors c, c, where c generated a single word w and c generated a syntactic pair (a, b).The first two in Theorem 1 are exactly the same as Theorem 2.2 in. Therefore we only need to prove the for p ([a, b] ) and p(w, [a, b] ). Here the last step used Lemma 10. Since both Z and Z a can be bounded by O(n), and DISPLAYFORM0 is bounded by (4κ + √ 2K) 2, we know the first term is of order Ω(1/n 2), and the second term is negligible. We end with the proof of Lemma 2.Proof of Lemma 2. Just for this proof, we use the following notation. Let I d×d be the d-dimensional identity matrix, and let x 1, x 2,..., x n be i.i.d. draws from N (0, I d×d). Let y i = x i 2, and note that y, v i, c) ). We first cover the unit sphere by a finite number of metric balls of small radius. Then we show that with high probability, the partition function at the center of these balls is indeed bounded below by a constant. Finally, we show that the partition function evaluated at an arbitrary point on the unit sphere can't be too far from the partition function at one of the ball centers provided the norms of the v i are not too large. We finish by appropriately controlling the norms of the v i. ≥.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a generative model for compositional word embeddings that captures syntactic relations, and provide empirical verification and evaluation.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: Building deep reinforcement learning agents that can generalize and adapt to unseen environments remains a fundamental challenge for AI. This paper describes progresses on this challenge in the context of man-made environments, which are visually diverse but contain intrinsic semantic regularities. We propose a hybrid model-based and model-free approach, LEArning and Planning with Semantics (LEAPS), consisting of a multi-target sub-policy that acts on visual inputs, and a Bayesian model over semantic structures. When placed in an unseen environment, the agent plans with the semantic model to make high-level decisions, proposes the next sub-target for the sub-policy to execute, and updates the semantic model based on new observations. We perform experiments in visual navigation tasks using House3D, a 3D environment that contains diverse human-designed indoor scenes with real-world objects. LEAPS outperforms strong baselines that do not explicitly plan using the semantic content. Deep reinforcement learning (DRL) has undoubtedly witnessed strong achievements in recent years BID7; BID9. However, training an agent to solve tasks in a new unseen scenario, usually referred to as its generalization ability, remains a challenging problem . In model-free RL, the agent is trained to reactively make decisions from the observations, e.g., first-person view, via a black-box policy approximator. However the generalization ability of agents trained by model-free RL is limited, and is even more evident on tasks that require extensive planning BID9 ). On the other hand, model-based RL learns a dynamics model, predicting the next observation when taking an action. With the model, sequential decisions can be made via planning. However, learning a model for complex tasks and with high dimensional observations, such as images, is challenging. Current approaches for learning action-conditional models from video are only accurate for very short horizons BID3 ). Moreover, it is not clear how to efficiently adapt such models to changes in the domain. In this work, we aim to improve the generalization of RL agents in domains that involve highdimensional observations. Our insight is that in many realistic settings, building a pixel-accurate model of the dynamics is not necessary for planning high-level decisions. There are semantic structures and properties that are shared in real-world man-made environments. For example, rooms in indoor scenes are often arranged by their mutual functionality (e.g., bathroom next to bedroom, dining room next to kitchen). Similarly, objects in rooms are placed at locations of practical significance (e.g., nightstand next to bed, chair next to table). Humans often make use of such structural priors when exploring a new scene, or when making a high-level plan of actions in the domain. However, pixel-level details are still necessary for carrying out the high-level plan. For example, we need high-fidelity observations to locate and interact with objects, open doors, etc. Based on this observation, we propose a hybrid framework, LEArning and Planning with Semantics (LEAPS), which consists of a model-based component that works on the semantic level to pursue a high-level target, and a model-free component that executes the target by acting on pixel-level inputs. Concretely, we train model-free multi-target subpolicies in the form of neural networks that take the first-person views as input and sequentially execute sub-targets towards the final goal; build a semantic model in the form of a latent variable model that only takes semantic signals, i.e., low-dimensional binary vectors, as input and is dynamically updated to plan the next sub-target. LEAPS has following advantages: via model-based planning, generalization ability is improved; by learning the prior distribution of the latent variable model, we capture the semantic consistency among the environments; the semantic model can be efficiently updated by posterior inference when the agent is exploring the unseen environment, which is effective even with very few exploration experiences thanks to the Bayes rule; and the semantic model is lightweight and fully interpretable. Our approach requires observations that are composed of both pixel-level data and a list of semantic properties of the scene. In general, automatically extracting high-level semantic structure from data is difficult. As a first step, in this work we focus on domains where obtaining semantics is easy. In particular, we consider environments which resemble the real-world and have strong object detectors available . An example of such environments is House3D which contains 45k human-designed 3D scenes BID12. House3D provides a diverse set of scene layouts, object types, sizes and connectivity, which all conform to a consistent "natural" semantics. Within these complex scenes, we tackle navigation tasks within novel indoor scenes. Note that this problem is extremely challenging as the agent needs to reach far-away targets which can only be completed effectively if it can successfully reason about the overall structure of the new scenario. Lastly, we emphasize that although we consider navigation as a concrete example in this work, our approach is general and can be applied to other tasks for which semantic structures and signals are availableOur extensive experiments show that our LEAPS framework outperforms strong model-free RL approaches, even when the semantic signals are given as input to the policy. Furthermore, the relative improvements of LEAPS over baselines become more significant when the targets are further away from the agent's birthplace, indicating the effectiveness of planning on the learned semantic model. Most deep RL agents are tested in the same training environments , disregarding generalization. While limited, robust training approaches have been proposed to enforce an agent's generalization ability, such as domain randomization BID10 and data augmentation by generating random mazes for training . In our work, we use a test set of novel unseen environments, where an agent cannot resort to memorization or simple pattern matching to solve the task. Meta-learning has shown promising for fast adaptation to novel environments. Methods include learning a good initialization for gradient descent or learning a neural network that can adapt its policy during exploration BID2 ). We propose to learn a Bayesian model over the semantic level and infer the posterior structure via the Bayes rule. Our approach can work even without any exploration steps in a new environment and is interpretable and can be potentially combined with any graph-based planning algorithm. Our work can be viewed as a special case of hierarchical reinforcement learning (HRL). Unlike other approaches BID11 BID1, in our work high-level planning is performed based on the semantic signals. With orders of magnitudes fewer parameters, our approach is easier to learn compared to recurrent controllers. LEAPS assumes a discrete semantic signal in addition to the continuous state. A similar assumption is also adopted in BID13, where the discrete signals are called "attributes" and used for planning to solve compositional tasks within the same fully observable environment. BID5 use additional discrete signals to tackle the sparse reward problem. The schema network further assumes that even the continuous visual signal can be completely represented in a binary form and therefore directly runs logical reasoning on the binary states. For evaluating our approach, we focus on the problem of visual navigation, which has been studied extensively . Classical approaches build a 3D map of the scene using SLAM, which is subsequently used for planning . More recently, end-to-end approaches have been applied to tackle various domains, such as maze , indoor scenes BID14 and Google street view . Evidently, navigation performance deteriorates as the agent's distance from the target increases BID14 BID12. To aid navigation and boost performance, auxiliary tasks are often introduced during training. Another direction for visual navigation is to use a recurrent neural network and represent the memory in the form of a 2D spatial map (; ; BID9 BID14 such that a differentiable planning computation can be performed on the spatial memory. Our approach considers more general graph structures beyond dense 2D grids and captures relationships between semantic signals, which we utilize as an informative latent structure in semantically rich environments like House3D. Similar to our work, Savinov et al. BID6 constructs a graph of nodes corresponding to different locations of the environment. However, they rely on a pre-exploration step within the test scene and build the graph completely from the pixel space. In LEAPS, we use semantic knowledge and learn a prior over the semantic structures that are shared across real-world scenes. This allows us to directly start solving for the task at hand without any exploratory steps. We assume familiarity with standard DRL notations. Complete definitions are in Appendix A. We consider a contextual Markov decision process E(c) defined by E(c) = (S, A, P (s |s, a; c), r(s, a; c)). Here c represents the objects, layouts and any other semantic information describing the environment, and is sampled from C, the distribution of possible semantic scenarios. For example, c can be intuitively understood as encoding the complete map for navigation, or the complete object and obstacle layouts in robotics manipulations, not known to the agent in advance, and we refer to them as the context. Semantic Signal: At each time step, the agent observes from s a tuple (s o, s s), which consists of: a high-dimensional observation s o, e.g., the first person view image, and a low-dimensional discrete semantic signal s s, which encodes semantic information. Such signals are common in AI, e.g., in robotic manipulation tasks s s indicates whether the robot is holding an object; for games it is the game status of a player; in visual navigation it indicates whether the agent reached a landmark; while in the AI planning literature, s s is typically a list of predicates that describe binary properties of objects. We assume s s is provided by an oracle function, which can either be directly provided by the environment or extracted by some semantic extractor. Generalization: Let µ(a|{s (t) } t; θ) denote the agent's policy parametrized by θ conditioned on the previous states {s (t) } t. The objective of generalization is to train a policy on training environments E train such that the accumulative reward R(µ(θ); c) on test set E test is maximized. The key motivation of LEAPS is the fact that while each environment can be different in visual appearances, there are structural similarities between environments that can be captured as a probabilistic graphical model over the semantic information. On a high level, we aim to learn a Bayesian model M (D, c) that captures the semantic properties of the context c, from the agent's exploration experiences D. Given a new environment E(c), the agent computes the posterior P (c |D, M) for the unknown context c via the learned model M and its current experiences D. This allows the agent to plan according to its belief of c to reach the goal more effectively. Thanks to the Bayes rule, this formulation allows probabilistic inference even with limited (or even no) exploration experiences. Learning an accurate and complete Bayesian model M (D, c) can be challenging. We learn an approximate latent variable model M(y, z; ψ) parameterized by ψ with observation variable y and latent variable z that only depend on the semantic signal s s. Suppose we have K different semantic signals T 1,..., T K and s s ∈ {0, 1} K where s s (T k) denotes whether the kth signal T k (e.g., landmarks in navigation) is reached or not. Assuming T 1 is the final goal of the task, from any state s, we want to reach some final state s with s s (T 1) = 1. In this work, we consider navigation as a concrete example, which can be represented as reaching a state where a desired semantic signal becomes'true'. We exploit the fact that navigation to a target can be decomposed into reaching several way points on way to the target, and therefore can be guided by planning on the semantic signals, i.e., arrival at particular way points. Note that there can be 2K different values for s s. For efficient computation, we assume independence between different semantic signals T k: we use a binary variable z i,j to denote whether some state s with s s (T j) = 1 can be "directly reached", i.e., by a few exploration steps, from some state s with s s (T i) = 1, regardless of other signals T k ∈ {T i, T j}. In addition, we also assume reversibility, i.e., z i,j = z j,i, so only K(K − 1)/2 latent variables are needed. Before entering the unknown environment, the agent does not know the true value of z i,j, but holds some prior belief DISPLAYFORM0 is some parameter to be learned. After some exploration steps, the agent receives a noisy observation y i,j of z i,j, i.e., whether a state s with s s (T j) = 1 is reached. We define the observation model P (y i,j |z i,j) as follows: DISPLAYFORM1 At any time step, the agent hold an overall belief P (z|Y) of the semantic structure of the unknown environment, based on its experiences Y, namely the samples of y. Multi-target sub-policies: With our semantic model, we correspondingly learn multi-target subpolicies µ(a|{s DISPLAYFORM0 is particularly trained for sub-target T i, i.e., reaching a state s with s s (T i) = 1. Hence the semantic model can be treated as a modelbased planning module that picks an intermediate sub-target for the sub-policies to execute so that the final target T 1 can be reached with the highest probability. Inference and planning on M: We assume the agent explores the current environment for a short horizon of N steps and receives semantic signals s s,..., s s (N). Then we compute the bit-OR operation over these binary vectors DISPLAYFORM1. By the reversibility assumption, for T i and T j with B(T i) = B(T j) = 1, we know that T i and T j are "directly reachable" for each other, namely a sample of y i,j = 1, and otherwise y i,j = 0. Combining all the history samples of y and the current batch from B as Y, we can perform posterior inference P (z|Y) by the Bayes rule. By the independence assumption, we can individually compute the belief of each latent variable z i,j, denoted byẑ i,j = P (z i,j |Y i,j). Given the current beliefsẑ i,j, the current semantic signals s s and the goal T 1, we search for an optimal plan τ * = {τ 0, τ 1, . . ., τ m−1, τ m}, where τ i ∈ {1 . . . K} denotes an index of concepts and particularly τ m = 1, so that the joint belief along the path from some current signal to the goal is maximized: DISPLAYFORM2 After obtaining τ, we execute the sub-policy for the next sub-target T τ 1, and then repeatedly update the model and replan every N steps. The model parameters ψ have two parts: ψ prior for the prior of z and ψ obs for the noisy observation y. Note that ψ obs is related to the performance of the sub-policies µ(θ): if µ(θ) has a high success rate for reaching sub-targets, ψ obs should be low; when µ(θ) is poor, ψ obs should be higher (cf. Eq. ).Learning ψ prior: We learn ψ prior from E train. During training, for each pair of semantic signals T i and T j, we run random explorations from some state s with s(T i) = 1. If eventually we reach some state s with s (T j) = 1, we consider T i and T j are reachable and therefore a positive sample z i,j = 1; otherwise a negative sample z i,j = 0. Suppose Z denotes the samples we obtained for z from E train. We run maximum likelihood estimate for ψ prior by maximizing L MLE (ψ prior) = P (Z|ψ prior).Learning ψ obs: There is no direct supervision for ψ obs. However, we can evaluate a particular value of ψ obs by policy evaluation on the validation environments E valid. We optimize the accumulative reward DISPLAYFORM0, with the semantic model M (ψ). Analytically optimizing L valid is hard. Instead, we apply local search in practice to find the optimal ψ obs. The LEAPS agent consists of two parts, the multi-target sub-policy µ(T i, θ) and the semantic model M (ψ). Learning the multi-target sub-policies can be accomplished by any standard deep RL method on E train. For the semantic model, learning ψ prior does not depend on the sub-policies and can be reused even with different sub-policies; ψ obs depends on the sub-policies so it should be learned after µ(T i, θ) is obtained. Figure 1: Visualization of learned semantic prior of M(ψ): the most and least likely nearby rooms for dining room (L), bedroom (M) and outdoor (R), with numbers denoting ψ z, i.e., the probability of two rooms connecting to each other. Figure 2: Example of a successful trajectory. The agent is spawned inside the house, targeting "outdoor". Left: the 2D top-down map with sub-target trajectories ("outdoor" -orange; "garage" -blue; "living room" -green); Right, 1st row: RGB visual image; Right, 2nd row: the posterior of the semantic graph and the proposed sub-targets (red arrow). Initially, the agent starts by executing the sub-policy "outdoor" and then "garage" according to the prior knowledge (1st graph), but both fail (top orange and blue trajectories in the map). After updating its belief that garage and outdoor are not nearby (grey edges in the 2nd graph), it then executes the "living room" sub-policy with success (red arrow in the 2nd graph, green trajectory). Finally, it executes "outdoor" sub-policy again, explores the living room and reaches the goal (3rd graph, bottom orange trajectory). RoomNav is a concept driven navigation task based on the House3D environment BID12. In RoomNav, the agent is given a concept target, i.e., a room type, and needs to navigate to find the target room. RoomNav pre-selected a fixed set of target room types and provides a training set of 200 houses, a testing set of 50 houses and a small validation set of 20 houses. Semantic signals: We choose the K = 8 most common room types as our semantic signals, such that s s (T i) denotes whether the agent is currently in a room with type T i 1. When given a target T i, reaching a state s with s s (T i) = 1 becomes our final goal. House3D provides bounding boxes for rooms, which can be directly used as the oracle for semantic signals. But in practice, we only use these oracle signals to train a room type detector and use this detector to extract semantic information during evaluation. Details can be found in the beginning part of Sec. 6.The semantic model and sub-policies: In navigation, the reachability variable z i,j can naturally represent the connectivity between room type T i and room type T j 2. We run random explorations in training houses between rooms to collect samples for learning ψ prior. For learning ψ obs, we perform a grid search and evaluate on the validation set. For sub-policies, we learn target driven LSTM policies by A3C with shaped reward on E train. More details are in Appendix. G. In this section, we experiment on RoomNav and try to answer the following questions: Does the learned prior distribution capture meaningful semantic consistencies? Does our LEAPS agent generalize better than the model-freel RL agent that only takes image input? Our LEAPS Figure 3: Comparison in success rate with model-free baselines (Sec. 6.2). We evaluate performance of random policy (blue), model-free RL baseline (pure µ(θ), green) and our LEAPS agent (red), with increasing horizon H from left to right (left: H = 300; middle: H = 500; right: H = 1000). Each row shows a particular metric. Top row: success rate (y-axis) w.r.t. the distance in meters from the birthplace to target room (x-axis); middle row: success rate with confidence interval (yaxis) w.r.t. the shortest planning distance in the ground truth semantic model (x-axis); bottom row: relative improvement of LEAPS over the baseline (y-axis) w.r.t. the optimal plan distance (x-axis). As the number of planning computations, i.e., H/N, increases (from left to right), LEAPS agent outperforms baselines more. LEAPS also has higher relative improvements, i.e., 40% to 180%, for targets requiring more semantic planning computations (i.e., plan-steps > 2). agent takes additional semantic signals as input. How does LEAPS compare to other model-free RL approaches that also take the semantic signals as part of the inputs but in a different way from our semantic model? For example, what about replacing our semantic model with a complicated RNN controller? What do our LEAPS agents perform under some other metric considering the episode length? We consider a recently proposed metric, SPL BID0 in Sec. 6.4.Semantic Signals: All the semantic signals fed to our semantic model at test time are extracted by a CNN room type detector, which is trained on the (noisy) oracle semantic information provided by the House3D on E train and validated on E valid. During training, all the approaches directly use the oracle semantic signals. More details are in Appendix J and Appendix D. We visualize the learned prior P (z|ψ prior) in Fig. 1 with 3 room types and their most and least likely connected rooms. The learned prior indeed captures reasonable relationships: bathroom is likely to connect to a bedroom; kitchen is often near a dining room while garage is typically outdoor. We follow measure the testing success rate of different agents under various horizon lengths on E test. More details are in Appendix C and F. We compare our LEAPS agent with two baselines random policy (denoted by "random") and model-free RL agent that only takes in image input s o and executes µ(T i, θ) throughout the episode (denoted by "pure µ(θ)"). For LEAPS agent, we set N = 30, i.e., update the semantic model every 30 steps. We experiment on horizons H = 300, 500, 1000 and evaluate the success rate and relative improvements of our LEAPS agent over the baselines in Fig. 3. As the number of planning computations, H/N, increases, our LEAPS agent outperforms the baselines more significantly in success rate. Note that since targets of plan-steps 1 do not require any planning computations, hence it is as expected that LEAPS does not improve much over the pure policy. The best relative improvements are achieved for targets neither too faraway nor too close, i.e., plan steps equal to 3 or 4. Interestingly, we observe that there is a small success rate increase for targets that are 5 plan steps away. We suspect that this is because it is rare to see houses that has a diameter of 5 in the semantic model (imagine a house where you need to go through 5 rooms to reach a place). Such houses may have structural properties that makes navigation easier. Fig. 2 shows an example of a success trajectory of our LEAPS agent. We visualize the progression of the episode, describe the plans and show the updated graph after exploration. Here we consider two semantic-aware agents that also takes the semantic signals as input. We train new sub-policies µ s (θ s) taking both s o and s s as input. Note that updating and planning on M (Eq. 2) only depend on the current semantic signal s s, the target T i, and the accumulative bit-OR feature B. Hence, we fixed the same set of sub-policies µ(θ) used by our LEAPS agent, and train an LSTM controller with 50 hidden units on E train that takes all the necessary semantic information, and produce a sub-target every N steps. Training details are in Appendix I. Note that the only difference between our LEAPS agent and this HRL agent is the representation of the planning module. The LSTM controller has access to exactly the same semantic information as our model M and uses a much more complicated neural model. Thus we expect it to perform competitively to our LEAPS agent. The are shown in FIG1, where our LEAPS agent outperforms both baselines. The semantic augmented policy µ s (θ s) does not improve much on the original µ(θ). For the HRL agent with an LSTM controller, the LEAPS agent achieves higher relative improvements for targets requiring more planning computations (i.e., plan-steps > 1), and also has the following advantages: M can be learned more efficiently with much fewer parameters: an LSTM with 50 hidden units has over 10 4 parameters while M (ψ) only has 38 parameters 3; M can adapt to new sub-policies µ(θ) with little fine-tuning (ψ prior remains unchanged) while the LSTM controller needs to re-train; the model M and the planning procedure are fully interpretable. In the previous evaluation, we only consider the metric success rate under different horizons. Indeed, another informative metric will be the episode length. There are two important factors: we would expect a better navigation agent to finish the semantic task in a shorter amount of actions; we should assign more credits to the agent when it finishes a hard episode while less credits when finishing an easy one. Recently, there is a new evaluation metric proposed for embodied navigation agent by BID0 capturing both these two factors, i.e., the Success weighted by Path Length (SPL) metric. SPL is a function considering both success rate and the path length to reach the goal from the starting point defined by Li,Pi), where N is total episodes evaluated, S i indicates whether the episode is success or not, L i is the ground truth shortest path distance in the episode, P i is the number of steps the agent actually took. DISPLAYFORM0 We evaluate the performance of LEAPS agents against all baseline agents in the metric of both success rate and SPL in FIG2. Our LEAPS agent has the highest average SPL (rightmost column) with a big margin over all baseline agents in all the cases. Notably, the margin in SPL is much more significant than the margin in pure success rate. More importantly, as the horizon increases, namely, more planning computations allowed, the SPL margin of LEAPS over the best remaining 3 We assign the same value to all ψ We evaluate performance of the semantic augmented model-free agent ("aug. µ s (θ)", blue), the HRL agent with the same sub-policies as LEAPS but with an LSTM controller ("RNN control.", green) and our LEAPS agent (red), with increasing horizon H from left to right (left: H = 300; middle: H = 500; right: H = 1000). Top row: success rate (y-axis) w.r.t. the distance in meters from birthplace to target (x-axis); middle row: success rate with confidence interval (y-axis) w.r.t. the shortest planning distance in the ground truth semantic model (x-axis); bottom row: relative improvements of LEAPS over the baselines (y-axis) w.r.t. the optimal plan distance (x-axis). Our LEAPS agent outperforms both of the baselines for targets requring planning computations (i.e., plan-steps > 1). For faraway targets with plan-steps > 2 in longer horizons (H ≥ 500), LEAPS improves over augmented policy by 80% and over RNN controller by 10% in success rate. Note that even though the LSTM controller has two orders of magnitudes more parameters than our semantic model M, our LEAPS agent still performs better. baselines strictly increases. This again indicates the effectiveness of our semantic model and shows that it indeed helps solve harder tasks, i.e., finding those faraway targets requiring more planning computations. We also notice that in shorter horizons (H ≤ 500), for plan-steps equal to 4, LEAPS agents have the highest success rate but relatively lower SPL. This is because that our LEAPS agent updates the semantic model every fixed N = 30 steps. This relatively low update frequency may potentially increase the episode length to reach a goal that requires more planning computations. However, when the horizon is long, i.e., allowing enough planning computations, our LEAPS agents significantly outperform all the baselines in SPL metric. It will be helpful if the LEAPS agent can learn to update the semantic model instead of updating it in a fixed frequency. We leave this to future works. In this work, we proposed LEAPS to improve generalization of RL agents in unseen environments with diverse room layouts and object arrangements, while the underlying semantic information is opt plan-steps 1 2 3 4 5 overall Horizon H = 300 random 20.5 / 15.9 6.9 / 16.7 3.8 / 10.7 1.6 / 4.2 3.0 / 8.8 7.2 / 13.6 pure µ(θ) 49.4 / 47.6 11.8 / 27.6 2.0 / 4.8 2.6 / 10.8 4.2 / 13.2 13.1 / 22.9 aug.µ S (θ) 47.8 / 45.3 11.4 / 23.1 3.0 / 7.8 3.4 / 8.1 4.4 / 11.2 13.0 / 20.5 RNN control. 52.7 / 45.2 13.6 / 23.6 3.4 / 9.6 3.4 / 10.2 6.0 / 17.6 14.9 / 21.9 LEAPS 53.4 / 58.4 15.6 / 31.5 4.5 / 12.5 3.6 / 6.6 7.0 / 18.0 16.4 / 27.9 Horizon H = 500 random 21.9 / 16.9 9.3 / 18.3 5.2 / 12.1 3.6 / 6.1 4.2 / 9.9 9.1 / 15.1 pure µ(θ) 54.0 / 57.5 15.9 / 25.6 3.8 / 7.7 2.8 / 6.4 4.8 / 8.6 16.2 / 22.9 aug.µ S (θ) 54.1 / 51.8 15.5 / 26.5 4.6 / 8. Our LEAPS agents have the highest success rates for all the cases requiring planning computations, i.e., plan-steps larger than 1. For SPL metric, LEAPS agents have the highest overall SPL value over all baseline methods (rightmost column). More importantly, as the horizon increases, LEAPS agents outperforms best baselines more. LEAPS requires a relatively longer horizon for the best practical performances since the semantic model is updated every fixed N = 30 steps, which may potentially increase the episode length for short horizons. More discussions are in Sec. 6.4.shared with the environments in which the agent is trained on. We adopt a graphical model over semantic signals, which are low-dimensional binary vectors. During evaluation, starting from a prior obtained from the training set, the agent plans on model, explores the unknown environment, and keeps updating the semantic model after new information arrives. For exploration, sub-policies that focus on multiple targets are pre-trained to execute primitive actions from visual input. The semantic model in LEAPS is lightweight, interpretable and can be updated dynamically with little explorations. As illustrated in the House3D environment, LEAPS works well for environments with semantic consistencies -typical of realistic domains. On random environments, e.g., random mazes, LEAPS degenerates to exhaustive search. Our approach is general and can be applied to other tasks, such as robotics manipulations where semantic signals can be status of robot arms and object locations, or video games where we can plan on semantic signals such as the game status or current resources. In future work we will investigate models for more complex semantic structures. Environment: We consider a contextual Markov Decision Process E(c) defined by E(c) = (S, A, P (s |s, a; c), r(s, a; c)), where S is the state space and A is the action space. c represents the objects, layouts and any other semantic information describing the environment, and is sampled from C, the distribution of possible semantic scenarios. r(s, a; c) denotes the reward function while P (s |s, a; c) describes transition probability conditioned on c. For example, c can be intuitively understood as encoding the complete map for navigation, or the complete object and obstacle layouts in robotics manipulations, not known to the agent in advance, and we refer to them as the context. Semantic Signal: At each time step, the agent's observation is a tuple (s o, s s), which consists of: (a) a high-dimensional observation s o, e.g. the first person view image, and (b) a low-dimensional semantic signal s s, which encodes semantic information. Such low-dimensional discrete signals are commonly used in AI, e.g. in robotic manipulation tasks s s indicates whether the robot is holding an object; for games it is the game status of a player; in visual navigation it indicates whether the agent reached a landmark; while in the AI planning literature, s s is typically a list of predicates that describe binary properties of objects. We assume s s is provided by an oracle function, which can either be directly provided by the environment or extracted by some semantic extractor. Generalization: Let µ(a|{s (t) } t; θ) denote the agent's policy parametrized by θ conditioned on the previous states {s (t) } t and R(µ(θ); c) denote the accumulative reward of µ(θ) in E(c). The objective is to find the best policy that maximizes the expected accumulative reward DISPLAYFORM0 In practice, we sample a disjoint partition of a training set E train = {E(c i)} i and a testing set E test = {E(c j)} j, where {c i} and {c j} are samples from C. We train µ(θ) with a shaped reward r train only on E train, and measure the empirical generalization performance of the learned policy on E test with the original unshaped reward (e.g., binary reward of success or not). In RoomNav the 8 targets are: kitchen, living room, dining room, bedroom, bathroom, office, garage and outdoor. We inherit the success measure of "see" from BID12: the agent needs to see some corresponding object for at least 450 pixels in the input frame and stay in the target area for at least 3 time steps. For the binary signal s s, we obtain from the bounding box information for each room provided from SUNCG dataset BID8, which is very noisy. Originally the House3D environment supports 13 discrete actions. Here we reduce it to 9 actions: large forward, forward, left-forward, right-forward, large left rotate, large right rotate, left rotate, right rotate and stay still. We following the evaluation setup from BID12 and measure the success rate on E test over 5750 test episodes, which consists of 5000 random generated configurations and 750 specialized for faraway targets to increase the confidence of measured success rate. These 750 episodes are generated such that for each plan-distance, there are at least 500 evaluation episodes. Each test episode has a fixed configuration for a fair comparison between different approaches, i.e., the agent will always start from the same location with the same target in that episode. Note that we always ensure that the target is connected to the birthplace of the agent, and the the birthplace of the agent is never within the target room. In the experiment sections, we use a CNN detector to extract the semantic signals at test time. Here we also evaluate the performance of all the approaches when using the ground truth signal from the oracle provided by the House3D environment. The are in Figure 6, where we also include the LEAPS agent using CNN detector as a references. Generally, using both the ground truth signal and using the CNN detector yield comparable overall performances in both metrics of success rate and Figure 6: Metrics of Success Rate(%) / SPL(‰) evaluating the performances of LEAPS and baselines agents using the ground truth oracle semantic signals provided by the environments. We also include the performance of LEAPS agent using CNN detector as a reference. Note that even using an CNN detector, LEAPS agents outperforms all baselines in both metrics of success rate and SPL. Notably, the performance of LEAPS-CNN agents is comparable to LEAPS-true agents and sometimes even better. This indicates that our semantic model can indeed tolerate practical errors in CNN detectors. More discussions are in Sec. D. We illustrate the ground truth shortest distance information as well as the average episode length of success episodes for all the approaches. The are shown in FIG3. Note that the average ground truth shortest path is around 46.86 steps. Considering the fact agent has 9 actions per step as well as the strong partial observability, this indicates that our benchmark semantic navigation task is indeed challenging. For confidence interval of the measured success rate, we computed it by fitting a binomial distribution. For optimal plan steps, we firstly extract all the room locations, and then construct a graph where a vertex is a room while an edge between two vertices is the shortest distance between these two rooms. After obtaining the graph and a birthplace of the agent, we compute shortest path from the birthplace to the target on this graph to derive the optimal plan steps. Hyperparameters: We utilize the same policy architecture as BID12. It was mentioned in BID12 ) that using segmentation mask + depth signals as input leads to relatively better performances for policy learning. So we inherit this setting here. In the original House3D paper, a gated attention module is used to incorporate the target instruction. Here, since we only have K = 8 different sub-policies, we simply train an individual policy for each target and we empirically observe that this leads to better performances. We run A3C with γ = 0.97, batch size 64, learning rate 0.001 with Adam, weight decay 10 −5, entropy bonus 0.1. We backprop through at most 30 time steps. We also compute the squared l 2 norm of logits and added to the loss with a coefficient 0.01. We also normalize the advantage to mean 0 and standard deviation 1.Reward shaping: We used a shaped reward function similar to BID12: the reward at each time step is computed by the difference of shortest paths in meters from the agent's location to the goal after taking a action. We also add a time penalty of 0.1 and a collision penalty of 0.3. When the agent reaches the goal, the success reward is 10.Curriculum learning: We run a curriculum learning by increasing the maximum of distance between agent's birth meters and target by 3 meters every 10000 iterations. We totally run 60000 training iterations and use the final model as our learned policy µ(θ). After evalution on the validation set, we choose to run random exploration for 300 steps to collect a sample of z. For a particular environment, we collect totally 50 samples for each z i,j.For all i = j, we set ψ For the LSTM controller, we ran A2C with batch size 32, learning rate 0.001 with adam, weight decay 0.00001, gamma 0.99, entropy bonus 0.01 and advantage normalization. The reward function is designed as follows: for every subtask it propose, it gets a time penalty of 0.1; when the agent reach the target, it gets a success bonus of 2.The input of the LSTM controller consists of s s (t) (K bits), B (K bits), last subtask T k, and the final target T i. We convert T i and T k to a one-hot vector and combine the other two features to feed into the LSTM. Hence the input dimension of LSTM controller is 4K, namely 32 in RoomNav. For the semantic augmented LSTM policy, µ s (θ s), we firstly use the CNN extract visual features from s o and combine the input semantic features and the visual features as the combined input to the LSTM in the policy. We noticed that in order to have a room type classifier, only using the single first person view image is not enough. For example, the agent may face towards a wall, which is not informative, but is indeed inside the bedroom (and the bed is just behind).So we take the panoramic view as input, which consists of 4 images, s 1 o,..., s 4 o with different first person view angles. The only exception is that for target "outdoor", we notice that instead of using a panoramic view, simply keeping the recent 4 frames in the trajectory leads to the best prediction accuracy. We use an CNN feature extractor to extract features f (s i o) by applying CNN layers with kernel size 3, strides and channels. We also use relu activation and batch norm. Then we compute the attention weights over these 4 visual features by l i = f (s) and a i = softmax(l i). Then we compute the weighted average of these four frames g = i a i f (s i o) and feed it to a single layer perceptron with 32 hidden units. For each semantic signal, we generate 15k positive and 15k negative training data from E train and use Adam optimizer with learning rate 5e-4, weight decay 1e-5, batch size 256 and gradient clip of 5. We keep the model that has the best prediction accuracy on E valid.For a smooth prediction during testing, we also have a hard threshold and filtering process on the CNN outputs: s s (T i) will be 1 only if the output of CNN has confidence over 0.85 for consecutively 3 steps.

Represent the short summary for finding the corresponding scientific article (long)

We propose a hybrid model-based & model-free approach using semantic information to improve DRL generalization in man-made environments.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We present an agent that uses a beta-vae to extract visual features and an attention mechanism to ignore irrelevant features from visual observations to enable robust transfer between visual domains.