Question: Represent the scientific article to find a short TLDR

Question: State-of-the-art on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et. al. propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15-40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task. Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs) BID12, form an important building block for many tasks that require modeling of sequential data. RNNs have been successfully employed for several such tasks including language modeling BID23 BID24 BID25, speech recognition BID9 BID19, and machine translation BID34. RNNs make output predictions at each time step by computing a hidden state vector h t based on the current input token and the previous states. This sequential computation underlies their ability to map arbitrary input-output sequence pairs. However, because of their auto-regressive property of requiring previous hidden states to be computed before the current time step, they cannot benefit from parallelization. Variants of recurrent networks that use strided convolutions eschew the traditional time-step based computation BID15 BID20 BID4 BID7 BID6 BID16. However, in these models, the operations needed to learn dependencies between distant positions can be difficult to learn BID13 BID11. Attention mechanisms, often used in conjunction with recurrent models, have become an integral part of complex sequential tasks because they facilitate learning of such dependencies BID22 BID26 BID27 BID17.In BID33, the authors introduce the Transformer network, a novel architecture that avoids the recurrence equation and maps the input sequences into hidden states solely using attention. Specifically, the authors use positional encodings in conjunction with a multi-head attention mechanism. This allows for increased parallel computation and reduces time to convergence. The authors report for neural machine translation that show the Transformer networks achieves state-of-the-art performance on the WMT 2014 English-to-German and English-to-French tasks while being orders-of-magnitude faster than prior approaches. Transformer networks still require a large number of parameters to achieve state-of-the-art performance. In the case of the newstest2013 English-to-German translation task, the base model required 65M parameters, and the large model required 213M parameters. We propose a variant of the Transformer network which we call Weighted Transformer that uses self-attention branches in lieu of the multi-head attention. The branches replace the multiple heads in the attention mechanism of the original Transformer network, and the model learns to combine these branches during training. This branched architecture enables the network to achieve comparable performance at a significantly lower computational cost. Indeed, through this modification, we improve the state-of-the-art performance by 0.5 and 0.4 BLEU scores on the WMT 2014 English-to-German and English-to-French tasks, respectively. Finally, we present evidence that suggests a regularizing effect of the proposed architecture. Most architectures for neural machine translation (NMT) use an encoder and a decoder that rely on deep recurrent neural networks like the LSTM BID22 BID34 BID3. Several architectures have been proposed to reduce the computational load associated with recurrence-based computation BID7 BID6 BID15 BID16. Self-attention, which relies on dot-products between elements of the input sequence to compute a weighted sum BID21 BID26 BID17, has also been a critical ingredient in modern NMT architectures. The Transformer network BID33 avoids the recurrence completely and uses only self-attention. We propose a modified Transformer network wherein the multi-head attention layer is replaced by a branched self-attention layer. The contributions of the various branches is learned as part of the training procedure. The idea of multi-branch networks has been explored in several domains BID0 BID6 BID35. To the best of our knowledge, this is the first model using a branched structure in the Transformer network. In, the authors use a large network, with billions of weights, in conjunction with a sparse expert model to achieve competitive performance. BID0 analyze learned branching, through gates, in the context of computer vision while in BID6, the author analyzes a two-branch model with randomly sampled weights in the context of image classification. The original Transformer network uses an encoder-decoder architecture with each layer consisting of a novel attention mechanism, which the authors call multi-head attention, followed by a feedforward network. We describe both these components below. From the source tokens, learned embeddings of dimension d model are generated which are then modified by an additive positional encoding. The positional encoding is necessary since the network does not otherwise possess any means of leveraging the order of the sequence since it contains no recurrence or convolution. The authors use additive encoding which is defined as: DISPLAYFORM0 where pos is the position of a word in the sentence and i is the dimension of the vector. The authors also experiment with learned embeddings BID7 BID6 but found no benefit in doing so. The encoded word embeddings are then used as input to the encoder which consists of N layers each containing two sub-layers: (a) a multi-head attention mechanism, and (b) a feed-forward network. A multi-head attention mechanism builds upon scaled dot-product attention, which operates on a query Q, key K and a value V: DISPLAYFORM1 where d k is the dimension of the key. In the first layer, the inputs are concatenated such that each of (Q, K, V) is equal to the word vector matrix. This is identical to dot-product attention except for the scaling factor d k, which improves numerical stability. Multi-head attention mechanisms obtain h different representations of (Q, K, V), compute scaled dot-product attention for each representation, concatenate the , and project the concatenation with a feed-forward layer. This can be expressed in the same notation as Equation FORMULA1: DISPLAYFORM2 where the W i and W O are parameter projection matrices that are learned. Note that DISPLAYFORM3 where h denotes the number of heads in the multi-head attention. BID33 proportionally reduce DISPLAYFORM4 that the computational load of the multi-head attention is the same as simple self-attention. The second component of each layer of the Transformer network is a feed-forward network. The authors propose using a two-layered network with a ReLU activation. Given trainable weights W 1, W 2, b 1, b 2, the sub-layer is defined as: DISPLAYFORM5 The dimension of the inner layer is d f f which is set to 2048 in their experiments. For the sake of brevity, we refer the reader to BID33 for additional details regarding the architecture. For regularization and ease of training, the network uses layer normalization BID1 after each sub-layer and a residual connection around each full layer. Analogously, each layer of the decoder contains the two sub-layers mentioned above as well as an additional multi-head attention sub-layer that receives as inputs (V, K) from the output of the corresponding encoding layer. In the case of the decoder multi-head attention sub-layers, the scaled dot-product attention is masked to prevent future positions from being attended to, or in other words, to prevent illegal leftward-ward information flow. One natural question regarding the Transformer network is why self-attention should be preferred to recurrent or convolutional models. BID33 state three reasons for the preference: (a) computational complexity of each layer, (b) concurrency, and (c) path length between long-range dependencies. Assuming a sequence length of n and vector dimension d, the complexity of each layer is O(n 2 d) for self-attention layers while it is O(nd 2) for recurrent layers. Given that typically d > n, the complexity of self-attention layers is lower than that of recurrent layers. Further, the number of sequential computations is O for self-attention layers and O(n) for recurrent layers. This helps improved utilization of parallel computing architectures. Finally, the maximum path length between dependencies is O for the self-attention layer while it is O(n) for the recurrent layer. This difference is instrumental in impeding recurrent models' ability to learn long-range dependencies. We now describe the proposed architecture, the Weighted Transformer, which is more efficient to train and makes better use of representational power. In Equations FORMULA2 and, we described the attention layer proposed in BID33 comprising the multi-head attention sub-layer and a FFN sub-layer. For the Weighted Transformer, we propose a branched attention that modifies the entire attention layer in the Transformer network (including both the multi-head attention and the feed-forward network).The proposed attention layer can be mathematically described as: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 where M denotes the total number of branches, κ i, α i ∈ R + are learned parameters and W Oi ∈ R dv×dmodel. The FFN functions above are identical in form to Equation them, they have commensurately reduced dimensionality to ensure that no additional parameters are added to the network. Further, we require that κ i = 1 and α i = 1 so that Equation FORMULA8 is a weighted sum of the individual branch attention values. We now briefly contrast the modified architecture with the base Transformer model. In the same notation as-, the attention layer in the base model can be described as: DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 Instead of aggregating the contributions from the different heads through W O right away and using a feed-forward sub-layer, we retain head i for each of the M heads, learn to amplify or diminish their contribution, use a feed-forward sub-layer and then aggregate them, again in a learned fashion. In the equations above, κ can be interpreted as a learned concatenation weight and α as the learned addition weight. Indeed, κ scales the contribution of the various branches before α is used to sum them in a weighted fashion. We ensure that the simplex constraint is respected during each training step by projection. Finally, note that our modification does not add depth (i.e., through the FFN sublayers) to any of the attention head transformation since the feed-forward computation is merely split and not stacked. One interpretation of our proposed architecture is that it replaces the multi-head attention by a multibranch attention. Rather than concatenating the contributions of the different heads, they are instead treated as branches that a multi-branch network learns to combine. While it is possible that α and κ could be merged into one variable and trained, we found better training outcomes by separating them. It also improves the interpretability of the models gives that (α, κ) can be thought of as probability masses on the various branches. This mechanism adds O(M) trainable weights. This is an insignificant increase compared to the total number of weights. Indeed, in our experiments, the proposed mechanism added 192 weights to a model containing 213M weights already. Without these additional trainable weights, the proposed mechanism is identical to the multi-head attention mechanism in the Transformer. The proposed attention mechanism is used in both the encoder and decoder layers and is masked in the decoder layers as in the Transformer network. Similarly, the positional encoding, layer normalization, and residual connections in the encoder-decoder layers are retained. We eliminate these details from FIG0 for clarity. Instead of using (α, κ) learned weights, it is possible to also use a mixture-ofexperts normalization via a softmax layer. However, we found this to perform worse than our proposal. Unlike the Transformer, which weighs all heads equally, the proposed mechanism allows for ascribing importance to different heads. This in turn prioritizes their gradients and eases the optimization process. Further, as is known from multi-branch networks in computer vision BID6, such mechanisms tend to cause the branches to learn decorrelated input-output mappings. This reduces co-adaptation and improves generalization. This observation also forms the basis for mixture-ofexperts models. The weights κ and α are initialized randomly, as with the rest of the Transformer weights. In addition to the layer normalization and residual connections, we use label smoothing with ls = 0.1, attention dropout, and residual dropout with probability P drop = 0.1. Attention dropout randomly drops out elements BID31 from the softmax in.As in BID33, we used the Adam optimizer BID18 with (β 1, β 2) = (0.9, 0.98) and = 10 −9. We also use the learning rate warm-up strategy for Adam wherein the learning rate lr takes on the form: DISPLAYFORM0 for the all parameters except (α, κ) and DISPLAYFORM1 for (α, κ).This corresponds to the warm-up strategy used for the original Transformer network except that we use a larger peak learning rate for (α, κ) to compensate for their bounds. Further, we found that freezing the weights (κ, α) in the last 10K iterations aids convergence. During this time, we continue training the rest of the network. We hypothesize that this freezing process helps stabilize the rest of the network weights given the weighting scheme. We note that the number of iterations required for convergence to the final score is substantially reduced for the Weighted Transformer. We found that Weighted Transformer converges 15-40% faster as measured by the total number of iterations to achieve optimal performance. We train the baseline model for 100K steps for the smaller variant and 300K for the larger. We train the Weighted Transformer for the respective variants for 60K and 250K iterations. We found that the objective did not significantly improve by running it for longer. Further, we do not use any averaging strategies employed in BID33 and simply return the final model for testing purposes. In order to reduce the computational load associated with padding, sentences were batched such that they were approximately of the same length. All sentences were encoded using byte-pair encoding BID29 and shared a common vocabulary. Weights for word embeddings were tied to corresponding entries in the final softmax layer BID14 BID28. We trained all our networks on NVIDIA K80 GPUs with a batch containing roughly 25,000 source and target tokens. We benchmark our proposed architecture on the WMT 2014 English-to-German and English-toFrench tasks. The WMT 2014 English-to-German data set contains 4.5M sentence pairs. The English-to-French contains 36M sentence pairs. Transformer (small) BID33 27.3 38.1 Weighted Transformer (small) 28.4 38.9Transformer (large) BID33 28.4 41.0 Weighted Transformer (large) 28.9 41.4ByteNet BID16 23.7 -Deep-Att+PosUnk BID37 -39.2 GNMT+RL BID34 24.6 39.9 ConvS2S BID8 25.2 40.5 MoE 26.0 40.6 Table 1: Experimental on the WMT 2014 English-to-German (EN-DE) and English-toFrench (EN-FR) translation tasks. Our proposed model outperforms the state-of-the-art models including the Transformer BID33. The small model corresponds to configuration (A) in TAB1 while large corresponds to configuration (B).Results of our experiments are summarized in Table 1. The Weighted Transformer achieves a 1.1 BLEU score improvement over the state-of-the-art on the English-to-German task for the smaller network and 0.5 BLEU improvement for the larger network. In the case of the larger English-toFrench task, we note a 0.8 BLEU improvement for the smaller model and a 0.4 improvement for the larger model. Also, note that the performance of the smaller model for Weighted Transformer is close to that of the larger baseline model, especially for the English-to-German task. This suggests that the Weighted Transformer better utilizes available model capacity since it needs only 30% of the parameters as the baseline transformer for matching its performance. Our relative improvements do not hinge on using the BLEU scores for comparison; experiments with the GLEU score proposed in BID34 also yielded similar improvements. Finally, we comment on the regularizing effect of the Weighted Transformer. Given the improved , a natural question is whether the stem from improved regularization of the model. To investigate this, we report the testing loss of the Weighted Transformer and the baseline Transformer against the training loss in FIG1. Models which have a regularizing effect tend to have lower testing losses for the same training loss. We see this effect in our experiments suggesting that the proposed architecture may have better regularizing properties. This is not unexpected given similar outcomes for other branching-based strategies such as Shake-Shake Gastaldi FORMULA1 BID33 architecture and our proposed Weighted Transformer. Reported BLEU scores are evaluated on the English-to-German translation development set, newstest2013. Weighted Transformer 24.8 Train κ, α fixed to 1 24.5 Train α, κ fixed to 1 23.9 α, κ both fixed to 1 23.6 Without the simplex constraints 24.5 Table 3: Model ablations of Weighted Transformer on the newstest2013 English-to-German task for configuration (C). This shows that the learning both (α, κ) and retaining the simplex constraints are critical for its performance. In TAB1, we report sensitivity on the newstest2013 English-to-German task. Specifically, we vary the number of layers in the encoder/decoder and compare the performance of the Weighted Transformer and the Transformer baseline. Using the same notation as used in the original Transformer network, we label our configurations as (A), (B) and (C) with (C) being the smallest. The clearly demonstrate the benefit of the branched attention; for every experiment, the Weighted Transformer outperforms the baseline transformer, in some cases by up to 1.3 BLEU points. As in the case of the baseline Transformer, increasing the number of layers does not necessarily improve performance; a modest improvement is seen when the number of layers N is increased from 2 to 4 and 4 to 6 but the performance degrades when N is increased to 8. Increasing the number of heads from 8 to 16 in configuration (A) yielded an even better BLEU score. However, preliminary experiments with h = 16 and h = 32, like in the case with N, degrade the performance of the model. In Figure 3, we present the behavior of the weights (α, κ) for the second encoder layer of the configuration (C) for the English-to-German newstest2013 task. The figure shows that, in terms of relative weights, the network does prioritize some branches more than others; circumstantially by as much as 2×. Further, the relative ordering of the branches changes over time suggesting that the network is not purely exploitative. A purely exploitative network, which would learn to exploit a subset of the branches at the expense of the rest, would not be preferred since it would effectively reduce the number of available parameters and limit the representational power. Similar are seen for other layers, including the decoder layers; we omit them for brevity. Finally, we present an ablation study to highlight the ingredients of our proposal that assisted the improved BLEU score in Table 3. The show that having both α and κ as learned, in conjunction with the simplex constraint, was necessary for improved performance. Figure 3: Convergence of the (α, κ) weights for the second encoder layer of Configuration (C) for the English-to-German newstest2013 task. We smoothen the curves using a mean filter. This shows that the network does prioritize some branches more than others and that the architecture does not exploit a subset of the branches while ignoring others. Weights (α, κ) BLEU Learned 24.8 Random 21.1 Uniform 23.4 Table 4: Performance of the architecture with random and uniform normalization weights on the newstest2013 English-to-German task for configuration (C). This shows that the learned (α, κ) weights of the Weighted Transformer are crucial to its performance. The proposed modification can also be interpreted as a form of Shake-Shake regularization proposed in BID6. In this regularization strategy, random weights are sampled during forward and backward passes for weighing the various branches in a multi-branch network. During test time, they are weighed equally. In our strategy, the weights are learned instead of being sampled randomly. Consequently, no changes to the model are required during test time. In order to better understand whether the network benefits from the learned weights or if, at test time, random or uniform weights suffice, we propose the following experiment: the weights for the Weighted Transformer, including (α, κ) are trained as before, but, during test time, we replace them with (a) randomly sampled weights, and (b) 1/M where M is the number of incoming branches. In Table 4, we report experimental on the configuration (C) of the Weighted Transformer on the English-to-German newstest2013 data set (see TAB1 for details regarding the configuration). It is evident that random or uniform weights cannot replace the learned weights during test time. Preliminary experiments suggest that a Shake-Shake-like strategy where the weights are sampled randomly during training also leads to inferior performance. In order to analyze whether a hard (discrete) choice through gating will outperform our normalization strategy, we experimented with using gates instead of the proposed concatenation-addition strategy. Specifically, we replaced the summation in Equation FORMULA8 by a gating structure that sums up the contributions of the top k branches with the highest probabilities. This is similar to the sparselygated mixture of experts model in. Despite significant hyper-parameter tuning of k and M, we found that this strategy performs worse than our proposed mechanism by a large margin. We hypothesize that this is due to the fact that the number of branches is low, typically less than 16. Hence, sparsely-gated models lose representational power due to reduced capacity in the model. We plan to investigate the setup with a large number of branches and sparse gates in future work. We present the Weighted Transformer that trains faster and achieves better performance than the original Transformer network. The proposed architecture replaces the multi-head attention in the Transformer network by a multiple self-attention branches whose contributions are learned as a part of the training process. We report numerical on the WMT 2014 English-to-German and English-to-French tasks and show that the Weighted Transformer improves the state-of-the-art BLEU scores by 0.5 and 0.4 points respectively. Further, our proposed architecture trains 15 − 40% faster than the baseline Transformer. Finally, we present evidence suggesting the regularizing effect of the proposal and emphasize that the relative improvement in BLEU score is observed across various hyper-parameter settings for both small and large models.

represent this text to find a much much longer article about it with say hundreds-thousands of words:

For instance, <<We employ graph neural networks in the variational EM framework for efficient inference and learning of Markov Logic Networks.>>  to "Markov Logic Networks (MLNs), which elegantly combine logic rules and probabilistic graphical models, can be used to address many knowledge graph problems. However, inference in MLN is computationally intensive, making the industrial-scale application of MLN very difficult. In recent years, graph neural networks (GNNs) have emerged as efficient and effective tools for large-scale graph problems. Nevertheless, GNNs do not explicitly incorporate prior logic rules into the models, and may require many labeled examples for a target task. In this paper, we explore the combination of MLNs and GNNs, and use graph neural networks for variational inference in MLN. We propose a GNN variant, named ExpressGNN, which strikes a nice balance between the representation power and the simplicity of the model. Our extensive experiments on several benchmark datasets demonstrate that ExpressGNN leads to effective and efficient probabilistic logic reasoning. Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval. Since knowledge graphs may contain incorrect, incomplete or duplicated records, additional processing such as link prediction, attribute classification, and record de-duplication is typically needed to improve the quality of knowledge graphs and derive new facts. Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs . The logic rules incorporate prior knowledge and allow MLNs to generalize in tasks with small amount of labeled data, while the graphical model formalism provides a principled framework for dealing with uncertainty in data. However, inference in MLN is computationally intensive, typically exponential in the number of entities, limiting the real-world application of MLN. Graph neural networks (GNNs) have recently gained increasing popularity for addressing many graph related problems effectively (; ; ;). However, the design and training procedure of GNNs do not explicitly take into account the prior knowledge in the form of logic rules. To achieve good performance, these models typically require sufficient labeled instances on specific end tasks . In this paper, we explore the combination of the best of both worlds, aiming for a method which is data-driven yet can exploit the prior knowledge encoded in logic rules. To this end, we design a simple variant of graph neural networks, named ExpressGNN, which can be efficiently trained in the variational EM framework for MLN. An overview of our method is illustrated in Fig. 1. ExpressGNN and the corresponding reasoning framework lead to the following desiderata: • Efficient inference and learning: ExpressGNN can be viewed as the inference network for MLN, which scales up MLN inference to much larger knowledge graph problems. • Combining logic rules and data supervision: ExpressGNN can leverage the prior knowledge encoded in logic rules, as well as the supervision from labeled data. • Compact and expressive model: ExpressGNN may have small number of parameters, yet it is sufficient to represent mean-field distributions in MLN. Statistical relational learning. There is an extensive literature relating the topic of logic reasoning, and here we only focus on the approaches that are most relevant to statistical relational learning on knowledge graphs. Logic rules can compactly encode the domain knowledge and complex dependencies. Thus, hard logic rules are widely used for reasoning in earlier attempts, such as expert systems and inductive logic programming . However, hard logic is very brittle and has difficulty in coping with uncertainty in both the logic rules and the facts in knowledge graphs. Later studies have explored to introduce probabilistic graphical model in the field of logic reasoning, seeking to combine the advantages of relational and probabilistic approaches. Representative works including Relational Markov Networks (RMNs;) and Markov Logic Networks (MLNs;) were proposed in this . Markov Logic Networks. MLNs have been widely studied due to the principled probabilistic model and effectiveness in a variety of reasoning tasks, including entity resolution (a), social networks , information extraction , etc. MLNs elegantly handle the noise in both logic rules and knowledge graphs. However, the inference and learning in MLNs is computationally expensive due to the exponential cost of constructing the ground Markov network and the NP-complete optimization problem. This hinders MLNs to be applied to industry-scale applications. Many works appear in the literature to improve the original MLNs in both accuracy and efficiency (b; ; ;). Nevertheless, to date, MLNs still struggle to handle large-scale knowledge bases in practice. Our framework ExpressGNN overcomes the scalability challenge of MLNs by efficient stochastic training algorithm and compact posterior parameterization with graph neural networks. Graph neural networks. Graph neural networks (GNNs; ;) can learn effective representations of nodes by encoding local graph structures and node attributes. Due to the compactness of model and the capability of inductive learning, GNNs are widely used in modeling relational data . Recently, proposed Graph Markov Neural Networks (GMNNs), which employs GNNs together with conditional random fields to learn object representations. These existing works are simply data-driven, and not able to leverage the domain knowledge or human prior encoded in logic rules. To the best of our knowledge, ExpressGNN is the first work that connects GNNs with first-order logic rules to combine the advantages of both worlds. Knowledge graph embedding. Another line of research for knowledge graph reasoning is in the family of knowledge graph embedding methods, such as TransE , NTN , DistMult , ComplEx , and RotatE . These methods design various scoring functions to model relational patterns for knowledge graph reasoning, which are very effective in learning the transductive embeddings of both entities and relations. However, these methods are not able to leverage logic rules, which can be crucial in some relational learning tasks, and have no consistent probabilistic model. Compared to these methods, ExpressGNN has consistent probabilistic model built in the framework, and can incorporate knowledge from logic rules. A recent concurrent work has proposed probabilistic Logic Neural Network (pLogicNet), which integrates knowledge graph embedding methods with MLNs with EM framework. Compared to pLogicNet which uses a flattened embedding table as the entity representation, our work can better capture the structure knowledge encoded in the knowledge graph with GNNs, and supplement the knowledge from logic formulae for the prediction task. Our method is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN part and the embedding part. Thus, pLogicNet can be viewed as a special case of our work with the embedding part only. 3 PRELIMINARY Knowledge Graph. A knowledge graph is a tuple In the language of first-order logic, entities are also called constants. For instance, a constant can be a person or an object. Relations are also called predicates. Each predicate is a logic function defined over C, i.e., r(·): C ×... × C → {0, 1}. In general, the arguments of predicates are asymmetric. For instance, for the predicate r(c, c):= L(c, c) (L for Like) which checks whether c likes c, the arguments c and c are not exchangeable. With a particular set of entities assigned to the arguments, the predicate is called a ground predicate, and each ground predicate ≡ a binary random variable, which will be used to define MLN. For a d-ary predicate, there are M d ways to ground it. We denote an assignment as a r. For instance, with a r = (c, c), we can simply write a ground predicate r(c, c) as r(a r). Each observed fact in knowledge bases is a truth value {0, 1} assigned to a ground predicate. For instance, a fact o can be [L(c, c) = 1]. The number of observed facts is typically much smaller than that of unobserved facts. We adopt the open-world paradigm and treat these unobserved facts ≡ latent variables. As a clearer representation, we express a knowledge base K by a bipartite graph G K = (C, O, E), where nodes on one side of the graph correspond to constants C and nodes on the other side correspond to observed facts O, which is called factor in this case. The set of T edges, E = {e 1, . . ., e T}, connect constants and the observed facts. More specifically, an edge e = (c, o, i) between node c and o exists, if the ground predicate associated with o uses c as an argument in its i-th argument position (Fig. 2). Markov Logic Networks. MLNs use logic formulae to define potential functions in undirected graphical models. A logic formula f (·): C ×... × C → {0, 1} is a binary function defined via the composition of a few predicates. For instance, a logic formula f (c, c) can be where ¬ is negation and the equivalence is established by De Morgan's law. Similar to predicates, we denote an assignment of constants to the arguments of a formula f as a f, and the entire collection of consistent assignments of constants as A f = {a 1 f, a 2 f, . . .}. A formula with constants assigned to all of its arguments is called a ground formula. Given these logic representations, MLN can be defined as a joint distribution over all observed facts O and unobserved facts H as where Z(w) is the partition function summing over all ground predicates and φ f (·) is the potential function defined by a formula f as illustrated in Fig. 2. One form of φ f (·) can simply be the truth value of the logic formula f. For instance, if the formula is f (c, c):= ¬S(c) ∨ ¬F(c, c) ∨ S(c), then φ f (c, c) can simply take value 1 when f (c, c) is true and 0 otherwise. Other more sophisticated φ f can also be designed, which have the potential to take into account complex entities, such as images or texts, but will not be the focus of this paper. The weight w f can be viewed as the confidence score of the formula f: the higher the weight, the more accurate the formula is. Difference between KG and MLN. We note that the graph topology of knowledge graphs and MLN can are very different, although MLN is defined on top of knowledge graphs. Knowledge graphs are typically very sparse, where the number of edges (observed relations) is typically linear in the number of entities. However, the graphs associated with MLN are much denser, where the number of nodes can be quadratic or more in the number of entities, and the number of edges (dependency between variables) is also high-order polynomials in the number of entities. In this section, we introduce the variational EM framework for MLN inference and learning, where we will use ExpressGNN as a key component (detailed in Sec. 5). Markov Logic Networks model the joint probabilistic distribution of all observed and latent variables, as defined in Eq. 1. This model can be trained by maximizing the log-likelihood of all the observed facts log P w (O). However, it is intractable to directly maximize the objective, since it requires to compute the partition function Z(w) and integrate over all variables O and H. We instead optimize the variational evidence lower bound (ELBO) of the data log-likelihood, as follows where Q θ (H | O) is a variational posterior distribution of the latent variables given the observed ones. The equality in Eq. 2 holds if the variational posterior Q θ (H|O) equals to the true posterior P w (H|O). We then use the variational EM algorithm to effectively optimize the ELBO. The variational EM algorithm consists of an expectation step (E-step) and a maximization step (M-step), which will be called in an alternating fashion to train the model: 1) In the E-step (Sec. 4.1), we infer the posterior distribution of the latent variables, where P w is fixed and Q θ is optimized to minimize the KL divergence between Q θ (H|O) and P w (H|O); 2) In the M-step (Sec. 4.2), we learn the weights of the logic formulae in MLN, where Q θ is fixed and P w is optimized to maximize the data log-likelihood. In the E-step, which is also known as the inference step, we are minimizing the KL divergence between the variational posterior distribution Q θ (H|O) and the true posterior distribution P w (H|O). The exact inference of MLN is computationally intractable and proven to be NP-complete . Therefore, we choose to approximate the true posterior with a mean-field distribution, since the mean-field approximation has been demonstrated to scale up large graphical models, such as latent Dirichlet allocation for modeling topics from large text corpus . In the mean-field variational distribution, each unobserved ground predicate r(a r) ∈ H is independently inferred as follows: where each factorized distribution Q θ (r(a r)) follows the Bernoulli distribution. We parameterize the variational posterior Q θ with deep learning models as our neural inference network. The design of the inference network is very important and has a lot of considerations, since we need a compact yet expressive model to accurately approximate the true posterior distribution. We employ graph neural networks with tunable embeddings as our inference network (detailed in Sec. 5), which can trade-off between the model compactness and expressiveness. With the mean-field approximation, L ELBO (Q θ, P w) defined in Eq. 2 can be reorganized as below: where w f is fixed in the E-step and thus the partition function Z(w) can be treated as a constant. We notice that the first term E Q θ (H|O) [log P w (O, H)] has the summation over all formulae and all possible assignments to each formula. Thus this double summation may involve a large number of terms. The second term E Q θ (H|O) [log Q θ (H|O)] is the sum of entropy of the variational posterior distributions Q θ (r(a r)), which also involves a large number of terms since the summation ranges over all possible latent variables. Typically, the number of latent facts in database is much larger than the number of observed facts. Thus, both terms in the objective function pose the challenge of intractable computational cost. To address this challenge, we sample mini-batches of ground formulae to break down the exponential summations by approximating it with a sequence of summations with a controllable number of terms. More specifically, in each optimization iteration, we first sample a batch of ground formulae. For each ground formula in the sampled batch, we compute the first term in Eq. 4 by taking the expectation of the corresponding potential function with respect to the posterior of the involved latent variables. The mean-field approximation enables us to decompose the global expectation over the entire MLN into local expectations over ground formulae. Similarly, for the second term in Eq. 4, we use the posterior of the latent variables in the sampled batch to compute a local sum of entropy. For tasks that have sufficient labeled data as supervision, we can add a supervised learning objective to enhance the inference network, as follows: This objective is complementary to the ELBO on predicates that are not well covered by logic rules but have sufficient observed facts. Therefore, the overall E-step objective function becomes: where λ is a hyperparameter to control the weight. This overall objective essentially combines the knowledge in logic rules and the supervision from labeled data. In the M-step, which is also known as the learning step, we are learning the weights of logic formulae in Markov Logic Networks with the variational posterior Q θ (H|O) fixed. The partition function Z(w) in Eq. 4 is not a constant anymore, since we need to optimize those weights in the M-step. There are exponential number of terms in the partition function Z(w), which makes it intractable to directly optimize the ELBO. To tackle this problem, we adopt the widely used pseudo-log-likelihood as an alternative objective for optimization, which is defined as: where MB r(ar) is the Markov blanket of the ground predicate r(a r), i.e., the set of ground predicates that appear in some grounding of a formula with r(a r). For each formula i that connects r(a r) to its Markov blanket, we optimize the formula weight w i by gradient descent, with the derivative: where y r(ar) = 0 or 1 if r(a r) is an observed fact, and y r(ar) = Q θ (r(a r)) otherwise. With the independence property of Markov Logic Networks, the gradients of the logic formulae weights can be efficiently computed on the Markov blanket of each variable. For the M-step, we design a different sampling scheme to make it computationally efficient. For each variable in the Markov blanket, we take the truth value if it's observed and draw a sample from the variational posterior Q θ if it's latent. In the M-step, the ELBO of a fully observed ground formula depends on the formula weight, thus we need to consider all the fully observed ground formulae. It is computationally intractable to use all possible ground predicates to compute the gradients in Eq. 8. To tackle this challenge, we simply consider all the ground formulae with at most one latent predicate, and pick up the ground predicate if its truth value determines the formula's truth value. Therefore, we keep a small subset of ground predicates, each of which can directly determine the truth value of a ground formula. Intuitively, this small subset contains all representative ground predicates, and makes good estimation of the gradients with much cheaper computational cost. Algorithm 1: GNN Initialize entity node: In the neural variational EM framework, the key component is the posterior model, or the inference network. We need to design the inference network that is both expressive and efficient to approximate the true posterior distribution. A recent concurrent work uses a flattened embedding table as the entity representation to model the posterior. However, such simple posterior model is not able to capture the structure knowledge encoded in the knowledge graph. We employ graph neural networks with tunable embeddings to design our inference network. We also investigate the expressive power of GNN from theoretical perspective, which justifies our design. Our inference network, named ExpressGNN, consists of three parts: the first part is a vanilla graph neural network (GNN), the second part uses tunable embeddings, and the third part uses the embeddings to define the variational posterior. For simplicity, we assume that each predicate has two arguments (i.e., consider only r(c, c)). We design each part as follows: • We build a GNN on the knowledge graph G K, which is much smaller than the ground graph of MLN (see comparison in Fig. 2). The computational graph of the GNN is given in Algorithm 1. The GNN parameters θ 1 and θ 2 are shared across the entire graph and independent of the number of entities. Therefore, the GNN is a compact model with • For each entity in the knowledge graph, we augment its GNN embedding with a tunable embedding The tunable embeddings increase the expressiveness of the model. As there are M entities, the number of parameters in tunable embeddings is O(kM). • We use the augmented embeddings of c 1 and c 2 to define the variational posterior. Specifically, In summary, ExpressGNN can be viewed as a two-level encoding of the entities: the compact GNN assigns similar embeddings to similar entities in the knowledge graph, while the expressive tunable embeddings provide additional model capacity to encode entity-specific information beyond graph structures. The overall number of trainable parameters in ExpressGNN is O(d 2 + kM). By tuning the embedding size d and k, ExpressGNN can trade-off between the model compactness and expressiveness. For large-scale problems with a large number of entities (M is large), ExpressGNN can save a lot of parameters by reducing k. The combination of GNN and tunable embeddings makes the model sufficiently expressive to approximate the true posterior distributions. Here we provide theoretical analysis on the expressive power of GNN in the mean-field inference problem, and discuss the benefit of combining GNN and tunable embeddings in ExpressGNN. Recent studies show that the vanilla GNN embeddings can represent the of graph coloring, but fail to represent the of the more strict graph isomorphism check, i.e., GNN produces the same embedding for some nodes that should be distinguished. We first demonstrate this problem by a simple example: Example. • Entity A and B have opposite relations with E, i.e., F(A, E) = 1 versus F(B, E) = 0 in the knowledge graph, but running GNN on the knowledge graph will always produce the same embeddings for A and B, i.e., µ A = µ B. • L(A, E) and L(B, E) apparently have different posteriors. However, using GNN em- We can formally prove that solving the problem in the above example requires the graph embeddings to distinguish any non-isomorphic nodes in the knowledge graph. A formal statement is provided below (see Appendix E for the proof). Definition 5.1. Two ordered sequences of nodes (c 1, . . ., c n) and (c 1, . . ., c n) are isomorphic in a graph G K if there exists an isomorphism π: Implied by the theorem, to obtain an expressive enough representation for the posterior, we need a more powerful GNN variant. A recent work has proposed a powerful GNN variant , which can handle small graphs such as chemical compounds and protein structures, but it is computationally expensive due to the usage of high-dimensional tensors. As a simple yet effective solution, ExpressGNN augments the vanilla GNN with additional tunable embeddings, which is a trade-off between the compactness and expressiveness of the model. In summary, ExpressGNN has the following nice properties: • Efficiency: ExpressGNN directly works on the knowledge graph, instead of the huge MLN grounding graph, making it much more efficient than the existing MLN inference methods. • Compactness: The compact GNN model with shared parameters can be very memory efficient, making ExpressGNN possible to handle industry-scale problems. • Expressiveness: The GNN model can capture structure knowledge encoded in the knowledge graph. Meanwhile, the tunable embeddings can encode entity-specific information, which compensates for GNN's deficiency in distinguishing non-isomorphic nodes. • Generalizability: With the GNN embeddings, ExpressGNN may generalize to new entities or even different but related knowledge graphs unseen during training time without the need for retraining. Benchmark datasets. We evaluate ExpressGNN and other baseline methods on four benchmark datasets: UW-CSE , Cora , synthetic Kinship datasets, and FB15K-237 constructed from Freebase . Details and full statistics of the benchmark datasets are provided in Appendix B. General settings. We conduct all the experiments on a GPU-enabled (Nvidia RTX 2080 Ti) Linux machine powered by Intel Xeon Silver 4116 processors at 2.10GHz with 256GB RAM. We implement ExpressGNN using PyTorch and train it with Adam optimizer . To ensure a fair comparison, we allocate the same computational resources (CPU, GPU and memory) for all the experiments. We use the default tuned hyperparameters for competitor methods, which can reproduce the experimental reported in their original works. Model hyperparameters. For ExpressGNN, we use 0.0005 as the initial learning rate, and decay it by half for every 10 epochs without improvement of validation loss. For Kinship, UW-CSE and Cora, we run ExpressGNN with a fixed number of iterations, and use the smallest subset from the original split for hyperparameter tuning. For FB15K-237, we use the original validation set to tune the hyperparameters. We use a two-layer MLP with ReLU activation function as the nonlinear transformation for each embedding update step in the GNN model. We learn different MLP For each dataset, we search the configuration of ExpressGNN on either the validation set or the smallest subset. The configuration we search includes the embedding size, the split point of tunable embeddings and GNN embeddings, the number of embedding update steps, and the sampling batch size. For the inference experiments, the weights for all the logic formulae are fixed as 1. For the learning experiments, the weights are initialized as 1. For the choice of λ in the combined objective L θ in Eq. 6, we set λ = 0 for the inference experiments, since the query predicates are never seen in the training data and no supervision is available. For the learning experiments, we set λ = 1. We first evaluate the inference accuracy and efficiency of ExpressGNN. We compare our method with several strong MLN inference methods on UW-CSE, Cora and Kinship datasets. We also conduct ablation study to explore the trade-off between GNN and tunable embeddings. Experiment settings. For the inference experiments, we fix the weights of all logic rules as 1. A key advantage of MLN is that it can handle open-world setting in a consistent probabilistic framework. Therefore, we adopt open-world setting for all the experiments, as opposed to closed-world setting where unobserved facts (except the query predicates) are assumed to be false. We also report the performance under closed-world setting in Appendix C. Prediction tasks. The deductive logic inference task is to answer queries that typically involve single predicate. For example in UW-CSE, the task is to predict the AdvisedBy(c,c) relation for all persons in the set. In Cora, the task is to de-duplicate entities, and one of the query predicates is SameAuthor(c,c). As for Kinship, the task is to predict whether a person is male or female, i.e., Male(c). For each possible substitution of the query predicate with different entities, the model is tasked to predict whether it's true or not. Inference accuracy. The of inference accuracy on three benchmark datasets are reported in Table 1. A hyphen in the entry indicates that it is either out of memory or exceeds the time limit (24 hours). We denote our method as ExpressGNN-E since only the E-step is needed for the inference experiments. Note that since the lifted BP is guaranteed to get identical as BP , the of these two methods are merged into one row. For these experiments, ExpressGNN-E uses 64-dim GNN embeddings and 64-dim tunable embeddings. On Cora, all the baseline methods fail to handle the data scale under open-world setting, and ExpressGNN-E achieves good inference accuracy. On UW-CSE, ExpressGNN-E consistently outperforms all baselines. The Kinship dataset is synthesized and noise-free, and the number of entities increases linearly on the five sets S1-S5. HL-MRF achieves perfect accuracy for S1-S4, but is infeasible on the largest set S5. ExpressGNN-E yields similar but not perfect , which is presumably caused by the stochastic nature of our sampling and optimization procedure. Inference efficiency. The inference time corresponding to the experiments in Table 1 is summarized in Fig. 4. On UW-CSE (left table), ExpressGNN-E uses much shorter time for inference compared to all the baseline methods, and meanwhile ExpressGNN-E achieves the best inference performance. On Kinship (right figure), as the data size grows linearly from S1 to S5, the inference time of most baseline methods grows exponentially, while ExpressGNN-E maintains a nearly constant time cost, demonstrating its nice scalability. Some baseline methods such as MCMC and MC-SAT become infeasible for larger sets. HL-MRF maintains a comparatively short inference time, however, it has a huge increase of memory cost and is not able to handle the largest set S5. Ablation study. ExpressGNN can trade-off the compactness and expressiveness of model by tuning the dimensionality of GNN and tunable embeddings. We perform ablation study on the Cora dataset to investigate how this trade-off affects the inference accuracy. Results of different configurations of ExpressGNN-E are shown in Table 2. It is observed that GNN64+Tune4 has comparable performance with Tune64, but is consistently better than GNN64. Note that the number of parameters in GNN64+Tune4 is O(64 2 + 4|C|), while that in Tune64 is O(64|C|). When the number of entities is large, GNN64+Tune4 has much less parameters to train. This is consistent with our theoretical analysis : As a compact model, GNN saves a lot of parameters, but GNN alone is not expressive enough. A similar is observed for GNN64+Tune64 and Tune128. Therefore, ExpressGNN seeks a combination of two types of embeddings to possess the advantage of both: having a compact model and being expressive. The best configuration of their embedding sizes can be varied on different tasks, and determined by the goal: getting a portable model or better performance. We evaluate ExpressGNN in the knowledge base completion task on the FB15K-237 dataset, and compare it with state-of-the-art knowledge base completion methods. Experiment settings. To generate logic rules, we use Neural LP on the training set and pick up the candidates with top confidence scores. See Appendix D for examples of selected logic rules. We evaluate both inference-only and inference-and-learning version of ExpressGNN, denoted as ExpressGNN-E and ExpressGNN-EM, respectively. Prediction task. For each test query r(c, c) with respect to relation r, the model is tasked to generate a rank list over all possible instantiations of r and sort them according to the model's confidence on how likely this instantiation is true. Evaluation metrics. Following existing studies , we use filtered ranking where the test triples are ranked against all the candidate triples not appearing in the dataset. Candidate triples are generated by corrupting the subject or object of a query r(c, c). For evaluation, we compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10. Competitor methods. Since none of the aforementioned MLN inference methods can scale up to this dataset, we compare ExpressGNN with a number of state-of-the-art methods for knowledge base completion, including Neural Tensor Network (NTN;), Neural LP , DistMult , ComplEx , TransE , RotatE and pLogicNet. The of MLN and pLogicNet are directly taken from the paper. For all the other baseline methods, we use publicly available code with the provided best hyperparameters to run the experiments. Performance analysis. The experimental on the full training data are reported in Table 3 (100% columns). Both ExpressGNN-E and ExpressGNN-EM significantly outperform all the baseline methods. With learning the weights of logic rules, ExpressGNN-EM achieves the best performance. Compared to MLN, ExpressGNN achieves much better performance since MLN only relies on the logic rules while ExpressGNN can also leverage the labeled data as additional supervision. Compared to knowledge graph embedding methods such as TransE and RotatE, ExpressGNN can leverage the prior knowledge in logic rules and outperform these purely data-driven methods. Data efficiency. We investigate the data efficiency of ExpressGNN and compare it with baseline methods. Following , we split the knowledge base into facts / training / validation / testing sets, and vary the size of the training set from 0% to 100% to feed the model with complete facts set for training. From Table 3, we see that ExpressGNN performs significantly better than the baselines on smaller training data. With more training data as supervision, data-driven baseline methods start to close the gap with ExpressGNN. This clearly shows the benefit of leveraging the knowledge encoded in logic rules when there data is insufficient for supervised learning. Zero-shot relational learning. In practical scenarios, a large portion of the relations in the knowledge base are long-tail, i.e., most relations may have only a few facts . Therefore, it is important to investigate the model performance on relations with insufficient training data. We construct a zero-shot learning dataset based on FB15K-237 by forcing the training and testing data to have disjoint sets of relations. Table 4 shows the . As expected, the performance of all the supervised relational learning methods drop to almost zero. This shows the limitation of such methods when coping with sparse long-tail relations. Neural LP is designed to handle new entities in the test set , but still struggles to perform well in zero-shot learning. In contrast, ExpressGNN leverages both the prior knowledge in logic rules and the neural relational embeddings for reasoning, which is much less affected by the scarcity of data on long-tail relations. Both variants of our framework (ExpressGNN-E and ExpressGNN-EM) achieve significantly better performance. This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning. ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework. ExpressGNN employs GNNs to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae. ExpressGNN is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN and the embedding part. Extensive experiments on multiple benchmark datasets demonstrates the effectiveness and efficiency of ExpressGNN. We provide more examples in this section to show that it is more than a rare case that GNN embeddings alone are not expressive enough. A.1 EXAMPLE 1 Now, we use another example in Fig. 7 to show that even when the local structures are the same, the posteriors can still be different, which is caused by the formulae.... … (C,H,G,M)... … B DATASET DETAILS For our experiments, we use the following benchmark datasets: • The social network dataset UW-CSE contains publicly available information of students and professors in the CSE department of UW. The dataset is split into five sets according to the home department of the entities. • The entity resolution dataset Cora consists of a collection of citations to computer science research papers. The dataset is also split into five subsets according to the field of research. • We introduce a synthetic dataset that resembles the popular Kinship dataset . The original dataset contains kinship relationships (e.g., Father, Brother) among family members in the Alyawarra tribe from Central Australia. The synthetic dataset closely resembles the original Kinship dataset but with a controllable number of entities. To generate a dataset with n entities, we randomly split n entities into two groups which represent the first and second generation respectively. Within each group, entities are grouped into a few sub-groups representing the sisterand brother-hood. Finally, entities from different sub-groups in the first generation are randomly coupled and a sub-group in the second generation is assigned to them as their children. To generate the knowledge base, we traverse this family tree, and record all kinship relations for each entity. We generate five kinship datasets (Kinship S1-S5) by linearly increasing the number of entities. • The knowledge base completion benchmark FB15K-237 is a generic knowledge base constructed from Freebase , which is designed to a more challenging variant of FB15K. More specifically, FB15K-237 is constructed by removing nearduplicate and inverse relations from FB15K. The dataset is split into training / validation / testing and we use the same split of facts from training as in prior work . The complete statistics of these datasets are shown in Table 5. Examples of logic formulae used in four benchmark datasets are listed in Table 7. In Sec. 6.1 we compare ExpressGNN with five probabilistic inference methods under open-world semantics. This is different from the original works, where they generally adopt the closed-world setting due to the scalability issues. More specifically, the original works assume that the predicates (except the ones in the query) observed in the knowledge base is closed, meaning for all instantiations For sanity checking, we also conduct these experiments with a closed-world setting. We found the summarized in Table 6 are close to those reported in the original works. This shows that we have a fair setup (including memory size, hyperparameters, etc.) for those competitor methods. Additionally, one can find that the AUC-PR scores compared to those (Table 1) under open-world setting are actually better. This is due to the way the datasets were originally collected and evaluated generally complies with the closed-world assumption. But this is very unlikely to be true for realworld and large-scale knowledge base such as Freebase and WordNet, where many true facts between entities are not observed. Therefore, in general, the open-world setting is much more reasonable, which we follow throughout this paper. We list some examples of logic formulae used in four benchmark datasets in Table 7. The full list of logic formulae is available in our source code repository. Note that these formulae are not necessarily as clean as being always true, but are typically true. For UW-CSE and Cora, we use the logic formulae provided in the original dataset. UW-CSE provides 94 hand-coded logic formulae, and Cora provides 46 hand-coded rules. For Kinship, we hand-code 22 first-order logic formulae. For FB15K-237, we first use Neural LP on the full data to generate candidate rules. Then we select the ones that have confidence scores higher than 90% of the highest scored formulae sharing the same target predicate. We also de-duplicate redundant rules that can be reduced to other rules by switching the logic variables. Finally, we have generated 509 logic formulae for FB15K-237. Theorem. Consider a knowledge base K = (C, R, O) and any r ∈ R. Two latent random variables X:= r(c 1, . . ., c n) and X:= r(c 1, . . ., c n) have the same posterior distribution in any MLN if and only if (c 1, · · ·, c n) Then we give the proof as follows. A logic formula f can be represented as a factor graph, G f = (C f, R f, E f), where nodes on one side of the graph is the set of distinct constants C f needed in the formula, while nodes on the other side is the set of predicates R f used to define the formula. The set of edges, E f, will connect constants to predicates or predicate negation. That is, an edge e = (c, r, i) between node c and predicate r exists, if the predicate r use constant c in its i-th argument. We note that the set of distinctive constants used in the definition of logic formula are templates where actual constant can be instantiated from C. An illustration of logic formula factor graph can be found in Fig. 8. Similar to the factor graph for the knowledge base, we also differentiate the type of edges by the position of the argument. Therefore, every single formula can be represented by a factor graph. We will construct a factor graph representation to define a particular formula, and show that the MLN induced by this formula will in different posteriors for r(c 1, . . ., c n) and r(c 1, . . ., c n). The factor graph for the formula is constructed in the following way (see Fig. 7 as an example of the ing formula constructed using the following steps): The proof of this claim is given at the end of this proof. (ii) Next, we use G * c1:n to define a formula f. We first initialize the definition of the formula value as f (c 1, . . ., c n,c 1, . . .,c n) = ∧ r(ar):r(ar) ∈ G * c1:n ⇒ r(c 1, . . ., c n). Then, we changer(ar) in this formula to the negation ¬r(ar) if the observed value ofr(ar) is 0 in G * c1:n. We have defined a formula f using the above two steps. Suppose the MLN only contains this formula f. Then the two nodes r(c 1, . . ., c n) and r(c 1, . . ., c n) in this MLN must be distinguishable. The reason is, in MLN, r(c 1, . . ., c n) is connected to a ground formula f (c 1, . . ., c n,c 1, . . .,c n), whose factor graph representation is G * c1:n ∪ r(c 1, . . ., c n). In this formula, all variables are observed in the knowledge base K except for r(c 1, . . ., c n) and and the observation set is O * c. The formula value is f (c 1, . . ., c n,c 1, . . .,c n) = (1 ⇒ r(c 1, . . ., c n)). Clarification: Eq. 10 is used to define a formula and c i in this equation can be replaced by other constants, while Eq. 11 represents a ground formula whose arguments are exactly c 1,..., c n,c 1,...,c n. Based on (Condition), there is NO formula f (c 1, . . ., c n,c 1, . . .,c n) that contains r(c 1, . . ., c n) has"

Using branched attention with learned combination weights outperforms the baseline transformer for machine translation tasks.

We compare graph RNNs and graph ConvNets, and we consider the most generic class of graph ConvNets with residuality.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Imitation learning provides an appealing framework for autonomous control: in many tasks, demonstrations of preferred behavior can be readily obtained from human experts, removing the need for costly and potentially dangerous online data collection in the real world. However, policies learned with imitation learning have limited flexibility to accommodate varied goals at test time. Model-based reinforcement learning (MBRL) offers considerably more flexibility, since a predictive model learned from data can be used to achieve various goals at test time. However, MBRL suffers from two shortcomings. First, the model does not help to choose desired or safe outcomes -- its dynamics estimate only what is possible, not what is preferred. Second, MBRL typically requires additional online data collection to ensure that the model is accurate in those situations that are actually encountered when attempting to achieve test time goals. Collecting this data with a partially trained model can be dangerous and time-consuming. In this paper, we aim to combine the benefits of imitation learning and MBRL, and propose imitative models: probabilistic predictive models able to plan expert-like trajectories to achieve arbitrary goals. We find this method substantially outperforms both direct imitation and MBRL in a simulated autonomous driving task, and can be learned efficiently from a fixed set of expert demonstrations without additional online data collection. We also show our model can flexibly incorporate user-supplied costs at test-time, can plan to sequences of goals, and can even perform well with imprecise goals, including goals on the wrong side of the road. Reinforcement learning (RL) algorithms offer the promise of automatically learning behaviors from raw sensory inputs with minimal engineering. However, RL generally requires online learning: the agent must collect more data with its latest strategy, use this data to update a model, and repeat. While this is natural in some settings, deploying a partially-trained policy on a real-world autonomous system, such as a car or robot, can be dangerous. In these settings the behavior must be learned offline, usually with expert demonstrations. How can we incorporate such demonstrations into a flexible robotic system, like an autonomous car? One option is imitation learning (IL), which can learn policies that stay near the expert's distribution. Another option is model-based RL (MBRL) BID8 BID2, which can use the data to fit a dynamics model, and can in principle be used with planning algorithms to achieve any user-specified goal at test time. However, in practice, model-based and model-free RL algorithms are vulnerable to distributional drift BID32 BID24: when acting according to the learned model or policy, the agent visits states different from those seen during training, and in those it is unlikely to determine an effective course of action. This is especially problematic when the data intentionally excludes adverse events, such as crashes. A model ignorant to the possibility of a crash cannot know how to prevent it. Therefore, MBRL algorithms usually require online collection and training BID6 BID12. Imitation learning algorithms use expert demonstration data and, despite similar drift shortcomings BID26, can sometimes learn effective policies without additional online data collection BID35. However, standard IL offers little task flexibility since it only predicts low-level behavior. While several works augmented IL with goal conditioning BID4 BID1, these goals must be specified in advance during training, and are typically simple (e.g., turning left or right).Figure 1: We apply our approach to navigation in CARLA BID5. Columns 1,2: Images depicting the current scene. The overhead image depicts a 50 m 2 area. Column 3: LIDAR input and goals are provided to our deep imitative trajectory model, and plans to the goals are computed under the model's likelihood objective, and colored according to their ranking under the objective, with red indicating the best plan. The red square indicates the chosen high-level goal, and the yellow cross indicates a point along our plan used as a setpoint for a PID controller. The LIDAR map is 100 m 2, and each goal is ≥20 m away from the vehicle. Column 4: Our model can incorporate arbitrary test-time costs, and use them to adjust its planning objective and plan ranking. Figure 2: A brief taxonomy of learning-based control methods. In our scenario, we avoid online data collection, specifically from the policy we seek to imitate. We structure our imitation learner with a model to make it flexible to new tasks at test time. We compare against other offline approaches (front face).The goal in our work is to devise a new algorithm that combines the advantages of IL and MBRL, affording both the flexibility to achieve new user-specified goals at test time and the ability to learn entirely from offline data. By learning a deep probabilistic predictive model from expert-provided data, we capture the distribution of expert behaviors without using manually designed reward functions. To plan to a goal, our method infers the most probable expert state trajectory, conditioned on the current position and reaching the goal. By incorporating a model-based representation, our method can easily plan to previously unseen user-specified goals while respecting rules of the road, and can be flexibly repurposed to perform a wide range of test-time tasks without any additional training. Inference with this model resembles trajectory optimization in model-based reinforcement learning, and learning this model resembles imitation learning. Our method's relationship to other work is illustrated in Fig. 2. We demonstrate our method on a simulated autonomous driving task (see FIG0 . A high-level route planner provides navigational goals, which our model uses to automatically generate plans that obey the rules of the road, inferred entirely from data. In contrast to IL, our method produces an interpretable distribution over trajectories and can follow a variety of goals without additional training. In contrast to MBRL, our method generates human-like behaviors without additional data collection or learning. In our experiments, our approach substantially outperforms both MBRL and IL: it can efficiently learn near-perfect driving through the static-world CARLA simulator from just 7,000 trajectories obtained from 19 hours of driving. We also show that our model can flexibly incorporate and achieve goals not seen during training, and is robust to errors in the high-level navigation system, even when the high-level goals are on the wrong side of the road. Videos of our are available. To learn robot dynamics that are not only possible, but preferred, we construct a model of expert behavior. We fit a probabilistic model of trajectories, q, to samples of expert trajectories drawn from an unknown distribution p. A probabilistic model is necessary because expert behavior is often stochastic and multimodal: e.g., choosing to turn either left or right at an intersection are both common decisions. Because an expert's behavior depends on their perception, we condition our model, q, on observations φ. In our application, φ includes LIDAR features χ ∈ R H×W ×C and a small window of previous positions s −τ :0 = {s −τ, . . ., s 0}, such that φ = {χ, s −τ :0}.By training q(s 1:T |φ) to forecast expert trajectories with high likelihood, we model the sceneconditioned expert dynamics, which can score trajectories by how likely they are to come from the expert. At test time, q(s 1:T |φ) serves as a learned prior over the set of undirected expert trajectories. To execute samples from this distribution is to imitate an expert driver in an undirected fashion. We first describe how we use the generic form of this model to plan, and then discuss our particular implementation in Section 2.2. Besides simply imitating the expert demonstrations, we wish to direct our agent to desired goals at test time, and have the agent reason automatically about the mid-level details necessary to achieve these goals. In general, we can define a driving task by a set of goal variables G. We will instantiate examples of G concretely after the generic goal planning derivation. The probability of a plan conditioned on the goal G is given as posterior distribution p(s 1:T |G, φ). Planning a trajectory under this posterior corresponds to MAP inference with prior q(s 1:T |φ) and likelihood p(G|s 1:T, φ). We briefly derive the MAP inference starting from the posterior maximization objective, which uses the learned Imitative Model to generate plans that achieve abstract goals: DISPLAYFORM0 Waypoint planning: One example of a concrete inference task is to plan towards a specific goal location, or waypoint. We can achieve this task by using a tightly-distributed goal likelihood function centered at the user's desired final state. This effectively treats a desired goal location, g T, as if it were a noisy observation of a future state, with likelihood p(G|s 1:T, φ) = N (g T |s T, I). The ing inference corresponds to planning the trajectory s 1:T to a likely point under the distribution N (g T |s T, I). We can also plan to successive states with DISPLAYFORM1 ) if the user (or program) wishes to specify the desired end velocity or acceleration when reached the final goal g T location FIG3. Alternatively, a route planner may propose a set of waypoints with the intention that the robot should reach any one of them. This is possible using a Gaussian mixture likelihood and can be useful if some of those waypoints along a route are inadvertently located at obstacles or potholes (Fig. 4).Waypoint planning leverages the advantage of conditional imitation learning: a user or program can communicate where they desire the agent to go without knowing the best and safest actions. The planning-as-inference procedure produces paths similar to how an expert would acted to reach the given goal. In contrast to black-box, model-free conditional imitation learning that regresses controls, our method produces an explicit plan, accompanied by an explicit score of the plan's quality. This provides both interpretability and an estimate of the feasibility of the plan. Costed planning: If the user desires more control over the plan, our model has the additional flexibility to accept arbitrary user-specified costs c at test time. For example, we may have updated knowledge of new hazards at test time, such as a given map of potholes (Fig. 4) or a predicted cost map. Given costs c(s i |φ), this can be treated by including an optimality variable C in G, where BID33 BID11. The goal log-likelihood is log p({g T, C = 1}|s 1:T, φ) = log N (g T |s T, I) + Figure 4: Imitative planning to goals subject to a cost at test time. The cost bumps corresponds to simulated "potholes," which the imitative planner is tasked with avoiding. The imitative planner generates and prefers routes that curve around the potholes, stay on the road, and respect intersections. Demonstrations of this behavior were never observed by our model. DISPLAYFORM2 The primary structural requirement of an Imitative Model is the ability to compute q(s 1:T |φ). The ability to also compute gradients ∇ s 1:T q(s 1:T |φ) enables gradient-based optimization for planning. Finally, the quality and efficiency of learning are important. One deep generative model for Imitation Learning is the Reparameterized Pushforward Policy (R2P2) BID23 ). R2P2's use of pushforward distributions BID16, employed in other invertible generative models BID21 BID3 allows it to efficiently minimize both false positives and false negatives (type I and type II errors) BID18. Optimization of KL(p, q), which penalizes mode loss (false negatives), is straightforward with R2P2, as it can evaluate q(s 1:T |φ). Here, p is the sampleable, but unknown, distribution of expert behavior. Reducing false positives corresponds to minimizing KL(q, p), which penalizes q heavily for generating bad DISPLAYFORM0 DISPLAYFORM1 end while 6: return s 1:T samples under p. As p is unknown, R2P2 first uses a spatial cost modelp to approximate p, which we can also use as c in our planner. The learning objective is KL(p, q) + βKL(q,p).Figure 5: Architecture of m t and σ t, modified from BID23 with permission. In R2P2, q(s 1:T |φ) is induced by an invertible, differentiable function: f (z; φ): R 2T → R 2T, which warps latent samples from a base distribution z ∼ q 0 = N (0, I 2T ×2T) to the output space over s 1:T. f embeds the evolution of learned discrete-time stochastic dynamics; each state is given by: DISPLAYFORM2 The m t ∈ R 2 and σ t ∈ R 2×2 are computed by expressive, nonlinear neural networks that observe previous states and LIDAR input. The ing trajectory distribution is complex and multimodal. We modified the RNN method described by BID23 and used LIDAR features χ = R 200×200×2, with χ ij representing a 2-bin histogram of points below and above the ground in 0.5 m 2 cells (Fig 5). We used T = 40 trajectories at 5Hz (8 seconds of prediction or planning), τ = 19. At test time, we use three layers of spatial abstractions to plan to a faraway destination, common to model-based (not end-to-end) autonomous vehicle setups: coarse route planning over a road map, path planning within the observable space, and feedback control to follow the planned path BID19 BID29. For instance, a route planner based on a conventional GPSbased navigation system might output waypoints at a resolution of 20 meters -roughly indicating the direction of travel, but not accounting for the rules of the road or obstacles. The waypoints are treated as goals and passed to the Imitative Planner (Algorithm 1), which then generates a path chosen according to the optimization in Eq. 1. These plans are fed to a low-level controller (we use a PID-controller) that follows the plan. In Fig. 6 we illustrate how we use our model in our application. Figure 6: Illustration of our method applied to autonomous driving. Our method trains an Imitative Model from a dataset of expert examples. After training, the model is repurposed as an Imitative Planner. At test time, a route planner provides waypoints to the Imitative Planner, which computes expert-like paths to each goal. The best plan chosen according to the planning objective, and provided to a low-level PID-controller in order to produce steering and throttle actions. Previous work has explored conditional IL for autonomous driving. Two model-free approaches were proposed by BID1, to map images to actions. The first uses three network "heads", each head only trained on an expert's left/straight/right turn maneuvers. The robot is directed by a route planner that chooses the desired head. Their second method input the goal location into the network, however, this did not perform as well. While model-free conditional IL can be effective given a discrete set of user directives, our model-based conditional IL has several advantages. Our model has flexibility to handle more complex directives post training, e.g. avoiding hazardous potholes (Fig. 4) or other costs, the ability to rank plans and goals by its objective, and interpretability: it can generate entire planned and unplanned (undirected) trajectories. Work by BID12 also uses multi-headed model-free conditional imitation learning to "warm start" a DDPG driving algorithm BID13. While warm starting hastens DDPG training, any subsequent DDPG post fine-tuning is inherently trial-and-error based, without guarantees of safety, and may crash during this learning phase. By contrast, our method never executes unlikely transitions w.r.t. expert behavior at training time nor at test time. Our method can also stop the car if no plan reaches a minimum threshold, indicating none are likely safe to execute. While our target setting is offline data collection, online imitation learning is an active area of research in the case of hybrid IL-RL BID25 BID31 and "safe" IL BID30 BID17 BID34. Although our work does not consider multiagent environments, several methods predict the behavior of other vehicles or pedestrians. Typically this involves recurrent neural networks combined with Gaussian density layers or generative models based on some context inputs such as LIDAR, images, or known positions of external agents BID28 BID37 BID7 BID14. However, none of these methods can evaluate the likelihood of trajectories or repurpose their model to perform other inference tasks. Other methods include inverse reinforcement learning to fit a probabilistic reward model to human demonstrations using the principle of maximum entropy BID36 BID27 BID22. We evaluate our method using the CARLA urban driving simulator BID5. Each test episode begins with the vehicle randomly positioned on a road in the Town01 or Town02 maps. The task is to drive to a goal location, chosen to be the furthest road location from the vehicle's initial position. As shown in Fig. 6, we use three layers of spatial abstractions to plan to the goal location, common to model-based (not end-to-end) autonomous vehicle setups: coarse route planning over a road map, path planning within the observable space, and feedback control to follow the planned path BID19 BID29. First, we compute a route to the goal location using A * given knowledge of the road graph. Second, we set waypoints along the route no closer than 20 m of the vehicle at any time to direct the vehicle. Finally, we use a PID-controller to compute the vehicle steering value. The PID-controller was tuned to steer the vehicle towards a setpoint (target) 5 meters away along the planned path. We consider four metrics for this task: 1) Success rate in driving to the goal location without any collisions. 2) Proportion of time spent driving in the correct lane. 3) Frequency of crashes into obstacles. 4) Passenger comfort, by comparing the distribution of accelerations (and higher-order terms) between each method. To contrast the benefits of our method against existing approaches, we compare against several baselines that all receive the same inputs and training data as our method. Since our approach bridges model-free IL and MBRL, we include an IL baseline algorithm, and a MBRL baseline algorithm. PID control: The PID baseline uses the PID-controller to follow the high-level waypoints along the route. This corresponds to removing the middle layer of autonomous vehicle decision abstraction, which serves as a baseline for the other methods. The PID controller is effective when the setpoint is several meters away, but fails when the setpoint is further away (i.e. at 20 m), causing the vehicle to cut corners at intersections. We designed an IL baseline to control the vehicle. A common straightforward approach to IL is behavior-cloning: learning to predict the actions taken by a demon-strator BID20 BID0 BID15 BID1. Our setting is that of goal-conditioned IL: in order to achieve different behaviors, the imitator is tasked with generating controls after observing a target high-level waypoint and φ. We designed two baselines: one with the branched architecture of BID1, where actions are predicted based on left/straight/right "commands" derived from the waypoints, and other that predicts the setpoint for the PID-controller. Each receives the same φ and is trained with the same set of trajectories as our main method. We found the latter method very effective for stable control on straightaways. When the model encounters corners, however, prediction is more difficult, as in order to successfully avoid the curbs, the model must implicitly plan a safe path. In the latter method, we used a network architecture nearly identical to our approach's.. To compare against a purely model-based reinforcement learning algorithm, we propose a model-predictive control baseline. This baseline first learns a forwards dynamics model f: (s t−3, s t−2, s t−1, s t, a t) → s t+1 given observed expert data (a t are recorded vehicle actions). We use an MLP with two hidden layers, each 100 units. Note that our forwards dynamics model does not imitate the expert preferred actions, but only models what is physically possible. Together with the same LIDAR map χ our method uses to locate obstacles, this baseline uses its dynamics model to plan a reachability tree BID9 through the free-space to the waypoint while avoiding obstacles. We plan forwards over 20 time steps using a breadth-first search search over CARLA steering angle {−0.3, −0.1, 0., 0.1, 0.3}, noting valid steering angles are normalized to [−1, 1], with constant throttle at 0.5, noting the valid throttle range is. Our search expands each state node by the available actions and retains the 50 closest nodes to the waypoint. The planned trajectory efficiently reaches the waypoint, and can successfully plan around perceived obstacles to avoid getting stuck. To convert the LIDAR images into obstacle maps, we expanded all obstacles by the approximate radius of the car, 1.5 meters. Performance that compare our methods against baselines according to multiple metrics are includes in TAB1. With the exception of the success rate metric, lower numbers are better. We define success rate as the proportion of episodes where the vehicles navigated across the road map to a goal location on the other side without any collisions. In our experiments we do not include any other drivers or pedestrians, so a collision is w.r.t. a stationary obstacle. Collision impulse (in N · s) is the average cumulative collision intensities over episodes. " Wrong lane" and "Off road" percentage of the vehicle invading other lanes or offroad (averaged over time and episodes). While safety metrics are arguably the most important metric, passenger comfort is also relevant. Passenger comfort can be ambiguous to define, so we simply record the second to sixth derivatives of the position vector with respect to time, respectively termed acceleration, jerk, snap, crackle, and pop. In TAB1 we note the 99th percentile of each statistic given all data collected per path planning method. Generally speaking, lower numbers correspond to a smoother driving experience. The poor performance of the PID baseline indicates that the high-level waypoints do not communicate sufficient information about the correct driving direction. Imitation learning achieves better levels of comfort than MBRL, but exhibits substantially worse generalization from the training data, since it does not reason about the sequential structure in the task. Model-based RL succeeds on most of the trials in the training environment, but exhibits worse generalization. Notably, it also scores much worse than IL in terms of staying in the right lane and maintaining comfort, which is consistent with our hypothesis: it is able to achieve the desired goals, but does not capture the behaviors in the data. Our method performs the best under all metrics, far exceeding the success and comfort metrics of imitation learning, and far exceeding the lane-obeyance and comfort metrics of MBRL. To further illustrate the capability of our method to incorporate test-time costs, we designed a pothole collision experiment. We simulated 2m-wide potholes in the environment by randomly inserting them in the cost map offset from each waypoint, distributed N (µ = [−15m, 2m], Σ = diag([1, 0.01])), (i.e. the mean is centered on the right side of the lane 15m before each waypoint). We ran our method that incorporates a test-time cost map of the simulated potholes, and compared to our method that did not incorporate the cost map (and thus had no incentive to avoid potholes). In addition to the other metrics, we recorded the number of collisions with potholes. In TAB2, we see that our method with cost incorporated achieved nearly perfect pothole avoidance, while still avoiding collisions with the environment. To do so, it drove closer to the centerline, and occasionally dipped into the opposite lane. Our model internalized obstacle avoidance by staying on the road, and demonstrated its flexibility to obstacles not observed during training. FIG4 shows an example of this behavior. As another test of our model's capability to stay in the distribution of demonstrated behavior, we designed a "decoy waypoints" experiment, in which half of the waypoints are highly perturbed versions of the other half, serving as distractions for our planner. The planner is tasked with planning to all of the waypoints under the Gaussian mixture likelihood. The perturbation distribution is N (0, σ = 8m): each waypoint is perturbed with a standard deviation of 8 meters. We observed the imitative model to be surprisingly robust to decoy waypoints. Examples of this robustness are shown in Fig. 8. One failure mode of this approach is when decoy waypoints lie on a valid off-route path at intersections, which temporarily confuses the planner about the best route. In TAB3, we report the success rate and the mean number of planning rounds for successful and failed episodes. These numbers indicate our method can execute dozens to hundreds of planning rounds without decoy waypoints derailing it. We also designed an experiment to test our method under systemic bias in the route planner. Our method is provided waypoints on the wrong side of the road. We model this by increasing the goal likelihood observation noise. After tuning the noise, we found our method to still be very effective at navigating, and report in TAB3. This further illustrates our method's tendency to stay near the distribution of expert behavior, as our expert never drove on the wrong side of the road. Our method with waypoints on wrong side, Town01 10 / 10 0.338% 0.002% Our method with waypoints on wrong side, Town02 7 / 10 3.159% 0.044% We proposed a method that combines elements of imitation learning and model-based reinforcement learning (MBRL). Our method first learns what preferred behavior is by fitting a probabilistic model to the distribution of expert demonstrations at training time, and then plans paths to achieve userspecified goals at test time while maintaining high probability under this distribution. We demonstrated several advantages and applications of our algorithm in autonomous driving scenarios. In the context of MBRL, our method mitigates the distributional drift issue by explicitly preferring plans that stay close to the expert demonstration data. This implicitly allows our method to enforce basic safety properties: in contrast to MBRL, which requires negative examples to understand the potential for adverse outcomes (e.g., crashes), our method automatically avoids such outcomes specifically because they do not occur (or rarely occur) in the training data. In the context of imitation learning, our method provides a flexible, safe way to generalize to new goals by planning, compared to prior work on black-box, model-free conditional imitation learning. Our algorithm produces an explicit plan within the distribution of preferred behavior accompanied with a score: the former offers interpretability, and the latter provides an estimate of the feasibility of the plan. We believe our method is broadly applicable in settings where expert demonstrations are available, flexibility to new situations is demanded, and safety is critical. Figure 8: Tolerating bad waypoints. The planner prefers waypoints in the distribution of expert behavior: on the road at a reasonable distance. Columns 1,2: Planning with 1 /2 decoy waypoints. Columns 3,4: Planning with all waypoints on the wrong side of the road.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Hybrid Vision-Driven Imitation Learning and Model-Based Reinforcement Learning for Planning, Forecasting, and Control

Represent the short summary for finding the corresponding scientific article (long)

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.