Question: Markov Logic Networks (MLNs), which elegantly combine logic rules and probabilistic graphical models, can be used to address many knowledge graph problems. However, inference in MLN is computationally intensive, making the industrial-scale application of MLN very difficult. In recent years, graph neural networks (GNNs) have emerged as efficient and effective tools for large-scale graph problems. Nevertheless, GNNs do not explicitly incorporate prior logic rules into the models, and may require many labeled examples for a target task. In this paper, we explore the combination of MLNs and GNNs, and use graph neural networks for variational inference in MLN. We propose a GNN variant, named ExpressGNN, which strikes a nice balance between the representation power and the simplicity of the model. Our extensive experiments on several benchmark datasets demonstrate that ExpressGNN leads to effective and efficient probabilistic logic reasoning. Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval. Since knowledge graphs may contain incorrect, incomplete or duplicated records, additional processing such as link prediction, attribute classification, and record de-duplication is typically needed to improve the quality of knowledge graphs and derive new facts. Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs . The logic rules incorporate prior knowledge and allow MLNs to generalize in tasks with small amount of labeled data, while the graphical model formalism provides a principled framework for dealing with uncertainty in data. However, inference in MLN is computationally intensive, typically exponential in the number of entities, limiting the real-world application of MLN. Graph neural networks (GNNs) have recently gained increasing popularity for addressing many graph related problems effectively (; ; ;). However, the design and training procedure of GNNs do not explicitly take into account the prior knowledge in the form of logic rules. To achieve good performance, these models typically require sufficient labeled instances on specific end tasks . In this paper, we explore the combination of the best of both worlds, aiming for a method which is data-driven yet can exploit the prior knowledge encoded in logic rules. To this end, we design a simple variant of graph neural networks, named ExpressGNN, which can be efficiently trained in the variational EM framework for MLN. An overview of our method is illustrated in Fig. 1. ExpressGNN and the corresponding reasoning framework lead to the following desiderata: • Efficient inference and learning: ExpressGNN can be viewed as the inference network for MLN, which scales up MLN inference to much larger knowledge graph problems. • Combining logic rules and data supervision: ExpressGNN can leverage the prior knowledge encoded in logic rules, as well as the supervision from labeled data. • Compact and expressive model: ExpressGNN may have small number of parameters, yet it is sufficient to represent mean-field distributions in MLN. Statistical relational learning. There is an extensive literature relating the topic of logic reasoning, and here we only focus on the approaches that are most relevant to statistical relational learning on knowledge graphs. Logic rules can compactly encode the domain knowledge and complex dependencies. Thus, hard logic rules are widely used for reasoning in earlier attempts, such as expert systems and inductive logic programming . However, hard logic is very brittle and has difficulty in coping with uncertainty in both the logic rules and the facts in knowledge graphs. Later studies have explored to introduce probabilistic graphical model in the field of logic reasoning, seeking to combine the advantages of relational and probabilistic approaches. Representative works including Relational Markov Networks (RMNs;) and Markov Logic Networks (MLNs;) were proposed in this . Markov Logic Networks. MLNs have been widely studied due to the principled probabilistic model and effectiveness in a variety of reasoning tasks, including entity resolution (a), social networks , information extraction , etc. MLNs elegantly handle the noise in both logic rules and knowledge graphs. However, the inference and learning in MLNs is computationally expensive due to the exponential cost of constructing the ground Markov network and the NP-complete optimization problem. This hinders MLNs to be applied to industry-scale applications. Many works appear in the literature to improve the original MLNs in both accuracy and efficiency (b; ; ;). Nevertheless, to date, MLNs still struggle to handle large-scale knowledge bases in practice. Our framework ExpressGNN overcomes the scalability challenge of MLNs by efficient stochastic training algorithm and compact posterior parameterization with graph neural networks. Graph neural networks. Graph neural networks (GNNs; ;) can learn effective representations of nodes by encoding local graph structures and node attributes. Due to the compactness of model and the capability of inductive learning, GNNs are widely used in modeling relational data . Recently, proposed Graph Markov Neural Networks (GMNNs), which employs GNNs together with conditional random fields to learn object representations. These existing works are simply data-driven, and not able to leverage the domain knowledge or human prior encoded in logic rules. To the best of our knowledge, ExpressGNN is the first work that connects GNNs with first-order logic rules to combine the advantages of both worlds. Knowledge graph embedding. Another line of research for knowledge graph reasoning is in the family of knowledge graph embedding methods, such as TransE , NTN , DistMult , ComplEx , and RotatE . These methods design various scoring functions to model relational patterns for knowledge graph reasoning, which are very effective in learning the transductive embeddings of both entities and relations. However, these methods are not able to leverage logic rules, which can be crucial in some relational learning tasks, and have no consistent probabilistic model. Compared to these methods, ExpressGNN has consistent probabilistic model built in the framework, and can incorporate knowledge from logic rules. A recent concurrent work has proposed probabilistic Logic Neural Network (pLogicNet), which integrates knowledge graph embedding methods with MLNs with EM framework. Compared to pLogicNet which uses a flattened embedding table as the entity representation, our work can better capture the structure knowledge encoded in the knowledge graph with GNNs, and supplement the knowledge from logic formulae for the prediction task. Our method is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN part and the embedding part. Thus, pLogicNet can be viewed as a special case of our work with the embedding part only. 3 PRELIMINARY Knowledge Graph. A knowledge graph is a tuple In the language of first-order logic, entities are also called constants. For instance, a constant can be a person or an object. Relations are also called predicates. Each predicate is a logic function defined over C, i.e., r(·): C ×... × C → {0, 1}. In general, the arguments of predicates are asymmetric. For instance, for the predicate r(c, c):= L(c, c) (L for Like) which checks whether c likes c, the arguments c and c are not exchangeable. With a particular set of entities assigned to the arguments, the predicate is called a ground predicate, and each ground predicate ≡ a binary random variable, which will be used to define MLN. For a d-ary predicate, there are M d ways to ground it. We denote an assignment as a r. For instance, with a r = (c, c), we can simply write a ground predicate r(c, c) as r(a r). Each observed fact in knowledge bases is a truth value {0, 1} assigned to a ground predicate. For instance, a fact o can be [L(c, c) = 1]. The number of observed facts is typically much smaller than that of unobserved facts. We adopt the open-world paradigm and treat these unobserved facts ≡ latent variables. As a clearer representation, we express a knowledge base K by a bipartite graph G K = (C, O, E), where nodes on one side of the graph correspond to constants C and nodes on the other side correspond to observed facts O, which is called factor in this case. The set of T edges, E = {e 1, . . ., e T}, connect constants and the observed facts. More specifically, an edge e = (c, o, i) between node c and o exists, if the ground predicate associated with o uses c as an argument in its i-th argument position (Fig. 2). Markov Logic Networks. MLNs use logic formulae to define potential functions in undirected graphical models. A logic formula f (·): C ×... × C → {0, 1} is a binary function defined via the composition of a few predicates. For instance, a logic formula f (c, c) can be where ¬ is negation and the equivalence is established by De Morgan's law. Similar to predicates, we denote an assignment of constants to the arguments of a formula f as a f, and the entire collection of consistent assignments of constants as A f = {a 1 f, a 2 f, . . .}. A formula with constants assigned to all of its arguments is called a ground formula. Given these logic representations, MLN can be defined as a joint distribution over all observed facts O and unobserved facts H as where Z(w) is the partition function summing over all ground predicates and φ f (·) is the potential function defined by a formula f as illustrated in Fig. 2. One form of φ f (·) can simply be the truth value of the logic formula f. For instance, if the formula is f (c, c):= ¬S(c) ∨ ¬F(c, c) ∨ S(c), then φ f (c, c) can simply take value 1 when f (c, c) is true and 0 otherwise. Other more sophisticated φ f can also be designed, which have the potential to take into account complex entities, such as images or texts, but will not be the focus of this paper. The weight w f can be viewed as the confidence score of the formula f: the higher the weight, the more accurate the formula is. Difference between KG and MLN. We note that the graph topology of knowledge graphs and MLN can are very different, although MLN is defined on top of knowledge graphs. Knowledge graphs are typically very sparse, where the number of edges (observed relations) is typically linear in the number of entities. However, the graphs associated with MLN are much denser, where the number of nodes can be quadratic or more in the number of entities, and the number of edges (dependency between variables) is also high-order polynomials in the number of entities. In this section, we introduce the variational EM framework for MLN inference and learning, where we will use ExpressGNN as a key component (detailed in Sec. 5). Markov Logic Networks model the joint probabilistic distribution of all observed and latent variables, as defined in Eq. 1. This model can be trained by maximizing the log-likelihood of all the observed facts log P w (O). However, it is intractable to directly maximize the objective, since it requires to compute the partition function Z(w) and integrate over all variables O and H. We instead optimize the variational evidence lower bound (ELBO) of the data log-likelihood, as follows where Q θ (H | O) is a variational posterior distribution of the latent variables given the observed ones. The equality in Eq. 2 holds if the variational posterior Q θ (H|O) equals to the true posterior P w (H|O). We then use the variational EM algorithm to effectively optimize the ELBO. The variational EM algorithm consists of an expectation step (E-step) and a maximization step (M-step), which will be called in an alternating fashion to train the model: 1) In the E-step (Sec. 4.1), we infer the posterior distribution of the latent variables, where P w is fixed and Q θ is optimized to minimize the KL divergence between Q θ (H|O) and P w (H|O); 2) In the M-step (Sec. 4.2), we learn the weights of the logic formulae in MLN, where Q θ is fixed and P w is optimized to maximize the data log-likelihood. In the E-step, which is also known as the inference step, we are minimizing the KL divergence between the variational posterior distribution Q θ (H|O) and the true posterior distribution P w (H|O). The exact inference of MLN is computationally intractable and proven to be NP-complete . Therefore, we choose to approximate the true posterior with a mean-field distribution, since the mean-field approximation has been demonstrated to scale up large graphical models, such as latent Dirichlet allocation for modeling topics from large text corpus . In the mean-field variational distribution, each unobserved ground predicate r(a r) ∈ H is independently inferred as follows: where each factorized distribution Q θ (r(a r)) follows the Bernoulli distribution. We parameterize the variational posterior Q θ with deep learning models as our neural inference network. The design of the inference network is very important and has a lot of considerations, since we need a compact yet expressive model to accurately approximate the true posterior distribution. We employ graph neural networks with tunable embeddings as our inference network (detailed in Sec. 5), which can trade-off between the model compactness and expressiveness. With the mean-field approximation, L ELBO (Q θ, P w) defined in Eq. 2 can be reorganized as below: where w f is fixed in the E-step and thus the partition function Z(w) can be treated as a constant. We notice that the first term E Q θ (H|O) [log P w (O, H)] has the summation over all formulae and all possible assignments to each formula. Thus this double summation may involve a large number of terms. The second term E Q θ (H|O) [log Q θ (H|O)] is the sum of entropy of the variational posterior distributions Q θ (r(a r)), which also involves a large number of terms since the summation ranges over all possible latent variables. Typically, the number of latent facts in database is much larger than the number of observed facts. Thus, both terms in the objective function pose the challenge of intractable computational cost. To address this challenge, we sample mini-batches of ground formulae to break down the exponential summations by approximating it with a sequence of summations with a controllable number of terms. More specifically, in each optimization iteration, we first sample a batch of ground formulae. For each ground formula in the sampled batch, we compute the first term in Eq. 4 by taking the expectation of the corresponding potential function with respect to the posterior of the involved latent variables. The mean-field approximation enables us to decompose the global expectation over the entire MLN into local expectations over ground formulae. Similarly, for the second term in Eq. 4, we use the posterior of the latent variables in the sampled batch to compute a local sum of entropy. For tasks that have sufficient labeled data as supervision, we can add a supervised learning objective to enhance the inference network, as follows: This objective is complementary to the ELBO on predicates that are not well covered by logic rules but have sufficient observed facts. Therefore, the overall E-step objective function becomes: where λ is a hyperparameter to control the weight. This overall objective essentially combines the knowledge in logic rules and the supervision from labeled data. In the M-step, which is also known as the learning step, we are learning the weights of logic formulae in Markov Logic Networks with the variational posterior Q θ (H|O) fixed. The partition function Z(w) in Eq. 4 is not a constant anymore, since we need to optimize those weights in the M-step. There are exponential number of terms in the partition function Z(w), which makes it intractable to directly optimize the ELBO. To tackle this problem, we adopt the widely used pseudo-log-likelihood as an alternative objective for optimization, which is defined as: where MB r(ar) is the Markov blanket of the ground predicate r(a r), i.e., the set of ground predicates that appear in some grounding of a formula with r(a r). For each formula i that connects r(a r) to its Markov blanket, we optimize the formula weight w i by gradient descent, with the derivative: where y r(ar) = 0 or 1 if r(a r) is an observed fact, and y r(ar) = Q θ (r(a r)) otherwise. With the independence property of Markov Logic Networks, the gradients of the logic formulae weights can be efficiently computed on the Markov blanket of each variable. For the M-step, we design a different sampling scheme to make it computationally efficient. For each variable in the Markov blanket, we take the truth value if it's observed and draw a sample from the variational posterior Q θ if it's latent. In the M-step, the ELBO of a fully observed ground formula depends on the formula weight, thus we need to consider all the fully observed ground formulae. It is computationally intractable to use all possible ground predicates to compute the gradients in Eq. 8. To tackle this challenge, we simply consider all the ground formulae with at most one latent predicate, and pick up the ground predicate if its truth value determines the formula's truth value. Therefore, we keep a small subset of ground predicates, each of which can directly determine the truth value of a ground formula. Intuitively, this small subset contains all representative ground predicates, and makes good estimation of the gradients with much cheaper computational cost. Algorithm 1: GNN Initialize entity node: In the neural variational EM framework, the key component is the posterior model, or the inference network. We need to design the inference network that is both expressive and efficient to approximate the true posterior distribution. A recent concurrent work uses a flattened embedding table as the entity representation to model the posterior. However, such simple posterior model is not able to capture the structure knowledge encoded in the knowledge graph. We employ graph neural networks with tunable embeddings to design our inference network. We also investigate the expressive power of GNN from theoretical perspective, which justifies our design. Our inference network, named ExpressGNN, consists of three parts: the first part is a vanilla graph neural network (GNN), the second part uses tunable embeddings, and the third part uses the embeddings to define the variational posterior. For simplicity, we assume that each predicate has two arguments (i.e., consider only r(c, c)). We design each part as follows: • We build a GNN on the knowledge graph G K, which is much smaller than the ground graph of MLN (see comparison in Fig. 2). The computational graph of the GNN is given in Algorithm 1. The GNN parameters θ 1 and θ 2 are shared across the entire graph and independent of the number of entities. Therefore, the GNN is a compact model with • For each entity in the knowledge graph, we augment its GNN embedding with a tunable embedding The tunable embeddings increase the expressiveness of the model. As there are M entities, the number of parameters in tunable embeddings is O(kM). • We use the augmented embeddings of c 1 and c 2 to define the variational posterior. Specifically, In summary, ExpressGNN can be viewed as a two-level encoding of the entities: the compact GNN assigns similar embeddings to similar entities in the knowledge graph, while the expressive tunable embeddings provide additional model capacity to encode entity-specific information beyond graph structures. The overall number of trainable parameters in ExpressGNN is O(d 2 + kM). By tuning the embedding size d and k, ExpressGNN can trade-off between the model compactness and expressiveness. For large-scale problems with a large number of entities (M is large), ExpressGNN can save a lot of parameters by reducing k. The combination of GNN and tunable embeddings makes the model sufficiently expressive to approximate the true posterior distributions. Here we provide theoretical analysis on the expressive power of GNN in the mean-field inference problem, and discuss the benefit of combining GNN and tunable embeddings in ExpressGNN. Recent studies show that the vanilla GNN embeddings can represent the of graph coloring, but fail to represent the of the more strict graph isomorphism check, i.e., GNN produces the same embedding for some nodes that should be distinguished. We first demonstrate this problem by a simple example: Example. • Entity A and B have opposite relations with E, i.e., F(A, E) = 1 versus F(B, E) = 0 in the knowledge graph, but running GNN on the knowledge graph will always produce the same embeddings for A and B, i.e., µ A = µ B. • L(A, E) and L(B, E) apparently have different posteriors. However, using GNN em- We can formally prove that solving the problem in the above example requires the graph embeddings to distinguish any non-isomorphic nodes in the knowledge graph. A formal statement is provided below (see Appendix E for the proof). Definition 5.1. Two ordered sequences of nodes (c 1, . . ., c n) and (c 1, . . ., c n) are isomorphic in a graph G K if there exists an isomorphism π: Implied by the theorem, to obtain an expressive enough representation for the posterior, we need a more powerful GNN variant. A recent work has proposed a powerful GNN variant , which can handle small graphs such as chemical compounds and protein structures, but it is computationally expensive due to the usage of high-dimensional tensors. As a simple yet effective solution, ExpressGNN augments the vanilla GNN with additional tunable embeddings, which is a trade-off between the compactness and expressiveness of the model. In summary, ExpressGNN has the following nice properties: • Efficiency: ExpressGNN directly works on the knowledge graph, instead of the huge MLN grounding graph, making it much more efficient than the existing MLN inference methods. • Compactness: The compact GNN model with shared parameters can be very memory efficient, making ExpressGNN possible to handle industry-scale problems. • Expressiveness: The GNN model can capture structure knowledge encoded in the knowledge graph. Meanwhile, the tunable embeddings can encode entity-specific information, which compensates for GNN's deficiency in distinguishing non-isomorphic nodes. • Generalizability: With the GNN embeddings, ExpressGNN may generalize to new entities or even different but related knowledge graphs unseen during training time without the need for retraining. Benchmark datasets. We evaluate ExpressGNN and other baseline methods on four benchmark datasets: UW-CSE , Cora , synthetic Kinship datasets, and FB15K-237 constructed from Freebase . Details and full statistics of the benchmark datasets are provided in Appendix B. General settings. We conduct all the experiments on a GPU-enabled (Nvidia RTX 2080 Ti) Linux machine powered by Intel Xeon Silver 4116 processors at 2.10GHz with 256GB RAM. We implement ExpressGNN using PyTorch and train it with Adam optimizer . To ensure a fair comparison, we allocate the same computational resources (CPU, GPU and memory) for all the experiments. We use the default tuned hyperparameters for competitor methods, which can reproduce the experimental reported in their original works. Model hyperparameters. For ExpressGNN, we use 0.0005 as the initial learning rate, and decay it by half for every 10 epochs without improvement of validation loss. For Kinship, UW-CSE and Cora, we run ExpressGNN with a fixed number of iterations, and use the smallest subset from the original split for hyperparameter tuning. For FB15K-237, we use the original validation set to tune the hyperparameters. We use a two-layer MLP with ReLU activation function as the nonlinear transformation for each embedding update step in the GNN model. We learn different MLP For each dataset, we search the configuration of ExpressGNN on either the validation set or the smallest subset. The configuration we search includes the embedding size, the split point of tunable embeddings and GNN embeddings, the number of embedding update steps, and the sampling batch size. For the inference experiments, the weights for all the logic formulae are fixed as 1. For the learning experiments, the weights are initialized as 1. For the choice of λ in the combined objective L θ in Eq. 6, we set λ = 0 for the inference experiments, since the query predicates are never seen in the training data and no supervision is available. For the learning experiments, we set λ = 1. We first evaluate the inference accuracy and efficiency of ExpressGNN. We compare our method with several strong MLN inference methods on UW-CSE, Cora and Kinship datasets. We also conduct ablation study to explore the trade-off between GNN and tunable embeddings. Experiment settings. For the inference experiments, we fix the weights of all logic rules as 1. A key advantage of MLN is that it can handle open-world setting in a consistent probabilistic framework. Therefore, we adopt open-world setting for all the experiments, as opposed to closed-world setting where unobserved facts (except the query predicates) are assumed to be false. We also report the performance under closed-world setting in Appendix C. Prediction tasks. The deductive logic inference task is to answer queries that typically involve single predicate. For example in UW-CSE, the task is to predict the AdvisedBy(c,c) relation for all persons in the set. In Cora, the task is to de-duplicate entities, and one of the query predicates is SameAuthor(c,c). As for Kinship, the task is to predict whether a person is male or female, i.e., Male(c). For each possible substitution of the query predicate with different entities, the model is tasked to predict whether it's true or not. Inference accuracy. The of inference accuracy on three benchmark datasets are reported in Table 1. A hyphen in the entry indicates that it is either out of memory or exceeds the time limit (24 hours). We denote our method as ExpressGNN-E since only the E-step is needed for the inference experiments. Note that since the lifted BP is guaranteed to get identical as BP , the of these two methods are merged into one row. For these experiments, ExpressGNN-E uses 64-dim GNN embeddings and 64-dim tunable embeddings. On Cora, all the baseline methods fail to handle the data scale under open-world setting, and ExpressGNN-E achieves good inference accuracy. On UW-CSE, ExpressGNN-E consistently outperforms all baselines. The Kinship dataset is synthesized and noise-free, and the number of entities increases linearly on the five sets S1-S5. HL-MRF achieves perfect accuracy for S1-S4, but is infeasible on the largest set S5. ExpressGNN-E yields similar but not perfect , which is presumably caused by the stochastic nature of our sampling and optimization procedure. Inference efficiency. The inference time corresponding to the experiments in Table 1 is summarized in Fig. 4. On UW-CSE (left table), ExpressGNN-E uses much shorter time for inference compared to all the baseline methods, and meanwhile ExpressGNN-E achieves the best inference performance. On Kinship (right figure), as the data size grows linearly from S1 to S5, the inference time of most baseline methods grows exponentially, while ExpressGNN-E maintains a nearly constant time cost, demonstrating its nice scalability. Some baseline methods such as MCMC and MC-SAT become infeasible for larger sets. HL-MRF maintains a comparatively short inference time, however, it has a huge increase of memory cost and is not able to handle the largest set S5. Ablation study. ExpressGNN can trade-off the compactness and expressiveness of model by tuning the dimensionality of GNN and tunable embeddings. We perform ablation study on the Cora dataset to investigate how this trade-off affects the inference accuracy. Results of different configurations of ExpressGNN-E are shown in Table 2. It is observed that GNN64+Tune4 has comparable performance with Tune64, but is consistently better than GNN64. Note that the number of parameters in GNN64+Tune4 is O(64 2 + 4|C|), while that in Tune64 is O(64|C|). When the number of entities is large, GNN64+Tune4 has much less parameters to train. This is consistent with our theoretical analysis : As a compact model, GNN saves a lot of parameters, but GNN alone is not expressive enough. A similar is observed for GNN64+Tune64 and Tune128. Therefore, ExpressGNN seeks a combination of two types of embeddings to possess the advantage of both: having a compact model and being expressive. The best configuration of their embedding sizes can be varied on different tasks, and determined by the goal: getting a portable model or better performance. We evaluate ExpressGNN in the knowledge base completion task on the FB15K-237 dataset, and compare it with state-of-the-art knowledge base completion methods. Experiment settings. To generate logic rules, we use Neural LP on the training set and pick up the candidates with top confidence scores. See Appendix D for examples of selected logic rules. We evaluate both inference-only and inference-and-learning version of ExpressGNN, denoted as ExpressGNN-E and ExpressGNN-EM, respectively. Prediction task. For each test query r(c, c) with respect to relation r, the model is tasked to generate a rank list over all possible instantiations of r and sort them according to the model's confidence on how likely this instantiation is true. Evaluation metrics. Following existing studies , we use filtered ranking where the test triples are ranked against all the candidate triples not appearing in the dataset. Candidate triples are generated by corrupting the subject or object of a query r(c, c). For evaluation, we compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10. Competitor methods. Since none of the aforementioned MLN inference methods can scale up to this dataset, we compare ExpressGNN with a number of state-of-the-art methods for knowledge base completion, including Neural Tensor Network (NTN;), Neural LP , DistMult , ComplEx , TransE , RotatE and pLogicNet. The of MLN and pLogicNet are directly taken from the paper. For all the other baseline methods, we use publicly available code with the provided best hyperparameters to run the experiments. Performance analysis. The experimental on the full training data are reported in Table 3 (100% columns). Both ExpressGNN-E and ExpressGNN-EM significantly outperform all the baseline methods. With learning the weights of logic rules, ExpressGNN-EM achieves the best performance. Compared to MLN, ExpressGNN achieves much better performance since MLN only relies on the logic rules while ExpressGNN can also leverage the labeled data as additional supervision. Compared to knowledge graph embedding methods such as TransE and RotatE, ExpressGNN can leverage the prior knowledge in logic rules and outperform these purely data-driven methods. Data efficiency. We investigate the data efficiency of ExpressGNN and compare it with baseline methods. Following , we split the knowledge base into facts / training / validation / testing sets, and vary the size of the training set from 0% to 100% to feed the model with complete facts set for training. From Table 3, we see that ExpressGNN performs significantly better than the baselines on smaller training data. With more training data as supervision, data-driven baseline methods start to close the gap with ExpressGNN. This clearly shows the benefit of leveraging the knowledge encoded in logic rules when there data is insufficient for supervised learning. Zero-shot relational learning. In practical scenarios, a large portion of the relations in the knowledge base are long-tail, i.e., most relations may have only a few facts . Therefore, it is important to investigate the model performance on relations with insufficient training data. We construct a zero-shot learning dataset based on FB15K-237 by forcing the training and testing data to have disjoint sets of relations. Table 4 shows the . As expected, the performance of all the supervised relational learning methods drop to almost zero. This shows the limitation of such methods when coping with sparse long-tail relations. Neural LP is designed to handle new entities in the test set , but still struggles to perform well in zero-shot learning. In contrast, ExpressGNN leverages both the prior knowledge in logic rules and the neural relational embeddings for reasoning, which is much less affected by the scarcity of data on long-tail relations. Both variants of our framework (ExpressGNN-E and ExpressGNN-EM) achieve significantly better performance. This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning. ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework. ExpressGNN employs GNNs to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae. ExpressGNN is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN and the embedding part. Extensive experiments on multiple benchmark datasets demonstrates the effectiveness and efficiency of ExpressGNN. We provide more examples in this section to show that it is more than a rare case that GNN embeddings alone are not expressive enough. A.1 EXAMPLE 1 Now, we use another example in Fig. 7 to show that even when the local structures are the same, the posteriors can still be different, which is caused by the formulae.... … (C,H,G,M)... … B DATASET DETAILS For our experiments, we use the following benchmark datasets: • The social network dataset UW-CSE contains publicly available information of students and professors in the CSE department of UW. The dataset is split into five sets according to the home department of the entities. • The entity resolution dataset Cora consists of a collection of citations to computer science research papers. The dataset is also split into five subsets according to the field of research. • We introduce a synthetic dataset that resembles the popular Kinship dataset . The original dataset contains kinship relationships (e.g., Father, Brother) among family members in the Alyawarra tribe from Central Australia. The synthetic dataset closely resembles the original Kinship dataset but with a controllable number of entities. To generate a dataset with n entities, we randomly split n entities into two groups which represent the first and second generation respectively. Within each group, entities are grouped into a few sub-groups representing the sisterand brother-hood. Finally, entities from different sub-groups in the first generation are randomly coupled and a sub-group in the second generation is assigned to them as their children. To generate the knowledge base, we traverse this family tree, and record all kinship relations for each entity. We generate five kinship datasets (Kinship S1-S5) by linearly increasing the number of entities. • The knowledge base completion benchmark FB15K-237 is a generic knowledge base constructed from Freebase , which is designed to a more challenging variant of FB15K. More specifically, FB15K-237 is constructed by removing nearduplicate and inverse relations from FB15K. The dataset is split into training / validation / testing and we use the same split of facts from training as in prior work . The complete statistics of these datasets are shown in Table 5. Examples of logic formulae used in four benchmark datasets are listed in Table 7. In Sec. 6.1 we compare ExpressGNN with five probabilistic inference methods under open-world semantics. This is different from the original works, where they generally adopt the closed-world setting due to the scalability issues. More specifically, the original works assume that the predicates (except the ones in the query) observed in the knowledge base is closed, meaning for all instantiations For sanity checking, we also conduct these experiments with a closed-world setting. We found the summarized in Table 6 are close to those reported in the original works. This shows that we have a fair setup (including memory size, hyperparameters, etc.) for those competitor methods. Additionally, one can find that the AUC-PR scores compared to those (Table 1) under open-world setting are actually better. This is due to the way the datasets were originally collected and evaluated generally complies with the closed-world assumption. But this is very unlikely to be true for realworld and large-scale knowledge base such as Freebase and WordNet, where many true facts between entities are not observed. Therefore, in general, the open-world setting is much more reasonable, which we follow throughout this paper. We list some examples of logic formulae used in four benchmark datasets in Table 7. The full list of logic formulae is available in our source code repository. Note that these formulae are not necessarily as clean as being always true, but are typically true. For UW-CSE and Cora, we use the logic formulae provided in the original dataset. UW-CSE provides 94 hand-coded logic formulae, and Cora provides 46 hand-coded rules. For Kinship, we hand-code 22 first-order logic formulae. For FB15K-237, we first use Neural LP on the full data to generate candidate rules. Then we select the ones that have confidence scores higher than 90% of the highest scored formulae sharing the same target predicate. We also de-duplicate redundant rules that can be reduced to other rules by switching the logic variables. Finally, we have generated 509 logic formulae for FB15K-237. Theorem. Consider a knowledge base K = (C, R, O) and any r ∈ R. Two latent random variables X:= r(c 1, . . ., c n) and X:= r(c 1, . . ., c n) have the same posterior distribution in any MLN if and only if (c 1, · · ·, c n) Then we give the proof as follows. A logic formula f can be represented as a factor graph, G f = (C f, R f, E f), where nodes on one side of the graph is the set of distinct constants C f needed in the formula, while nodes on the other side is the set of predicates R f used to define the formula. The set of edges, E f, will connect constants to predicates or predicate negation. That is, an edge e = (c, r, i) between node c and predicate r exists, if the predicate r use constant c in its i-th argument. We note that the set of distinctive constants used in the definition of logic formula are templates where actual constant can be instantiated from C. An illustration of logic formula factor graph can be found in Fig. 8. Similar to the factor graph for the knowledge base, we also differentiate the type of edges by the position of the argument. Therefore, every single formula can be represented by a factor graph. We will construct a factor graph representation to define a particular formula, and show that the MLN induced by this formula will in different posteriors for r(c 1, . . ., c n) and r(c 1, . . ., c n). The factor graph for the formula is constructed in the following way (see Fig. 7 as an example of the ing formula constructed using the following steps): The proof of this claim is given at the end of this proof. (ii) Next, we use G * c1:n to define a formula f. We first initialize the definition of the formula value as f (c 1, . . ., c n,c 1, . . .,c n) = ∧ r(ar):r(ar) ∈ G * c1:n ⇒ r(c 1, . . ., c n). Then, we changer(ar) in this formula to the negation ¬r(ar) if the observed value ofr(ar) is 0 in G * c1:n. We have defined a formula f using the above two steps. Suppose the MLN only contains this formula f. Then the two nodes r(c 1, . . ., c n) and r(c 1, . . ., c n) in this MLN must be distinguishable. The reason is, in MLN, r(c 1, . . ., c n) is connected to a ground formula f (c 1, . . ., c n,c 1, . . .,c n), whose factor graph representation is G * c1:n ∪ r(c 1, . . ., c n). In this formula, all variables are observed in the knowledge base K except for r(c 1, . . ., c n) and and the observation set is O * c. The formula value is f (c 1, . . ., c n,c 1, . . .,c n) = (1 ⇒ r(c 1, . . ., c n)). Clarification: Eq. 10 is used to define a formula and c i in this equation can be replaced by other constants, while Eq. 11 represents a ground formula whose arguments are exactly c 1,..., c n,c 1,...,c n. Based on (Condition), there is NO formula f (c 1, . . ., c n,c 1, . . .,c n) that contains r(c 1, . . ., c n) has

We employ graph neural networks in the variational EM framework for efficient inference and learning of Markov Logic Networks.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Reinforcement learning (RL) methods achieved major advances in multiple tasks surpassing human performance. However, most of RL strategies show a certain degree of weakness and may become computationally intractable when dealing with high-dimensional and non-stationary environments. In this paper, we build a meta-reinforcement learning (MRL) method embedding an adaptive neural network (NN) controller for efficient policy iteration in changing task conditions. Our main goal is to extend RL application to the challenging task of urban autonomous driving in CARLA simulator. " Every living organism interacts with its environment and uses those interactions to improve its own actions in order to survive and increase" BID13. Inspired from animal behaviorist psychology, reinforcement learning (RL) is widely used in artificial intelligence research and refers to goal-oriented optimization driven by an impact response or signal BID30. Properly formalized and converted into practical approaches BID9, RL algorithms have recently achieved major progress in many fields as games BID18 BID28 and advanced robotic manipulations BID12 BID17 beating human performance. However, and despite several years of research and evolution, most of RL strategies show a certain degree of weakness and may become computationally intractable when dealing with high-dimensional and non-stationary environments BID34. More specifically, the industrial application of autonomous driving in which we are interested in this work, remains a highly challenging "unsolved problem" more than one decade after the promising 2007 DARPA Urban Challenge BID2 ). The origin of its complexity lies in the large variability inherent to driving task arising from the uncertainty of human behavior, diversity of driving styles and complexity of scene perception. An interpretation of the observed vulnerability due to learning environment changes has been provided in contextaware (dependence) research assuming that "concepts in the real world are not eternally fixed entities or structures, but can have a different appearance or definition or meaning in different contexts" BID36. There are several tasks that require context-aware adaptation like weather forecast with season or geography, speech recognition with speaker origins and control processes of industrial installations with climate conditions. One solution to cope with this variability is to imitate the behavior of human who are more comfortable with learning from little experience and adapting to unexpected perturbations. These natural differences compared to machine learning and specifically RL methods are shaping the current research intending to eschew the problem of data inefficiency and improve artificial agents generalization capabilities BID10. Tackling this issue as a multi-task learning problem BID3, meta-learning has shown promising and stands as one of the preferred frames to design fast adapting strategies BID25 BID23. It refers to learn-to-learn approaches that aim at training a model on a set of different but linked tasks and subsequently generalize to new cases using few additional examples BID7.In this paper we aim at extending RL application to the challenging task of urban autonomous driving in CARLA simulator. We build a meta-reinforcement learning (MRL) method where agent policies behave efficiently and flexibly in changing task conditions. We consolidate the approach robustness by integrating a neural network (NN) controller that performs a continuous iteration of policy evaluation and improvement. The latter allows reducing the variance of the policy-based RL and accelerating its convergence. Before embarking with a theoretical modeling of the proposed approach in section 3, we introduce in the next section metalearning and related work in order to better understand the current issues accompanying its application to RL settings. In the last section, we evaluate our method using CARLA simulator and discuss experimental . Generally, in order to acquire new skills, it is more useful to rely on previous experience than starting from scratch. Indeed, we learn how to learn across tasks requiring, each time, less data and trial-and-error effort to conquer further skills BID10. The term meta-learning that refers to learning awareness on the basis of prior experience was first cited by BID0 in the field of educational psychology. It consists in taking control of a learning process and guiding it in accordance with the context of a specific task. In machine learning research, meta-learning is not a new concept and displays many similarities with the above definition BID31 BID26 BID19. It assumes that rather than building a learning strategy on the basis of a single task, it will be more effective to train over a series of tasks sharing a set of similarities then generalize to new situations. By acquiring prior biases, meta-learning addresses models inaccuracies achieving fast adaptation from few additional data BID4. At an architectural level, the learning is operated at two scales: a base-level system is assigned to rapid learning within each task, and a meta (higher) level system uses previous one feedback for gradual learning across tasks BID35.One of the first contribution to meta-learning is the classical Algorithm Selection Problem (ASP) proposed by BID24 considering the relationship between problem characteristics and the algorithm suitable to solve it. Then based on the concept of ASP, the No Free Lunch (NFL) theorem BID38 demonstrated that the generalization performance of any learner across all tasks is equal to 0. The universal learner is consequently a myth and each algorithm performs well only on a set of tasks delimiting its area of expertise. ASP and NFL theorem triggered a large amount of research assigned to parameter and algorithm recommendation BID8 BID1 BID29 BID22. In this type of meta-learning, a meta-learner apprehend the relationship between data characteristics called meta-features and base-learners performance in order to predict the best model to solve a specific task. Various meta-learners have been used and generally consist of shallow algorithms like decision trees, k-Nearest Neighbors and Support Vector Machines BID32. Regarding meta-features, the most commonly used ones included statistical and information-theoretic parameters as well as land-marking and model-based extractors BID33 ).The recent regain of interest in neural network models and more specifically deep learning ing from the advent of large training datasets and computational resources allowed the resurgence of neural network Meta-learning BID14. Instead of requiring explicit task characteristics, the meta-level learns from the structure of base-models themselves. Neural networks are particularly suitable to this kind of transfer learning given their inner capabilities of data features abstraction and rule inductions reflected in their connection weights and biases. The typology of meta-learners developed so far includes recurrent models, metrics and optimizers with several areas of application in classification, regression and RL BID15.Meta-learning algorithms extended recently to the context of RL can be classified in two broad categories. A first set of methods implement a recurrent neural network (RNN) or its memory-augmented variant (LSTM) as the meta-learner. BID6 study RL optimization in the frame of a reinforcement learning problem (RL2) where policies are represented with RNNs that receive past rewards and actions, in addition to the usual inputs. The approach is evaluated on multi-armed bandits (MAB) and tabular Markov Decision Processes (MDPs). In BID35, Advantage Actor-Critic (A2C) algorithms with recurrence are trained using different architectures of LSTM (simple, convolutional and stacked). The experiments are conducted on bandits problems with increasing level of complexity (dependent/independent arms and restless).In the second category, the learner gradients are used for meta-learning. Such methods are task-agnostic and adaptable to any model trained with gradient-descent. The gradient-based strategy has been originally introduced by BID7 with their Model-Agnostic Meta-Learning (MAML) algorithm. It has been demonstrated efficient for different problem settings including gradient RL with neural network policies. MAML mainly aims at generating a model initialization sensitive to changes and reaching optimal on a new scenario after just few gradient updates. Meta-SGD BID15 uses stochastic gradient descent to meta-learn, besides a model initialization, the inner loop learning rate and the direction of weights update. In Reptile BID20, the authors design a first order approximation of MAML computationally less expensive than the original method which includes second order derivative of gradient. BID27 propose a probabilistic view of MAML for continuous adaptation in RL settings. A competitive multi-agent environment (RoboSumo) was designed to run iterated adaptation games for the approach testing. A major part of MRL papers have been evaluated either at a preliminary level of experimentation or on elementary tasks (2D navigation, simulated muJoCo robots and bandit problems). In this work we consider an application of gradient-based MRL in a more challenging dynamic environment involving realistic and complex sides of real world tasks, which is CARLA simulator for autonomous driving BID5. The proposed model consists of a MRL framework embedding an adaptive NN controller to tackle both the nonstationarity and high dimensionality issues inherent to autonomous driving environments in CARLA simulator. The RL task considered in this work is a Markov Decision Process (MDP) defined according to the tuple (S, A, p, r, γ, ρ 0, H) where S is the set of states, A is the set of actions, p(s t+1 |s t, a t) is the state transition distribution predicting the probability to reach a state s t+1 in the next time step given current state and action, r is a reward function, γ is the discount factor, ρ 0 is the initial state distribution and H the horizon. Consider the sum of expected rewards (return) from a trajectory τ (0,H−1) = (s 0, a 0, ..., s H−1, a H−1, s H). A RL setting aims at learning a policy π of parameters θ (either deterministic or stochastic) that maps each state s to an optimal action a maximizing the return R of the trajectory. DISPLAYFORM0 Following the discounted return expressed above, we can define a state value function V (s): S → R to measure the current state return estimated under policy π: DISPLAYFORM1 In order to optimize the parameterized policy π θ, we use gradient descents like in the family of REINFORCE algorithms BID37 updating the policy parameters θ in the direction: DISPLAYFORM2 We build an approach of NN meta-learning compatible with RL setting. Our contribution consists in combining a gradient-based meta-learner like in MAML BID7 to learn a generalizable model initialization and a NN controller for more robust and continuous adaptation. The agent policy π θ approximated by a convolutional neural network (CNN) is trained to quickly adapt to a new task through few standard gradient descents. Explicitly, this consists in finding an optimal initialization of parameters θ * allowing a few-shot generalization of the learned model. Given a batch of tasks T i sampled from p(T), the metaobjective is formulated as follows: DISPLAYFORM0 The MRL approach includes two levels of processing: the inner and the outer loops associated respectively to the base and meta-learning. In the inner loop, we start by reducing the disturbances characterizing policy based methods and induced by the score function R t. Indeed, complex domains with conflicting dynamics and high dimensional observations like autonomous driving yield a large amount of uncertainty. One flexible solution to reduce disturbances and accelerate learning convergence is policy iteration. Subsequently, we modify the RL scheme by integrating a step of policy evaluation and improvement that generates added bonuses to guide the agent towards new states. The policy evaluation is performed with temporal difference (TD) learning combining Monte Carlo method and dynamic programming BID30 to learn, with step size ω, the value function approximated by a CNN: DISPLAYFORM1 Where δ t is the multi-step TD error that consists in bootstrapping the sampled returns from the value function estimate: DISPLAYFORM2 Multi-step returns allow the agent to gather more information on the environment before calculating the error in the value function estimates. Subsequently, the improvement of the policy is performed through the replacement of the score function R t by the TD error δ t in the policy gradient: DISPLAYFORM3 For each sampled task T i, the policy parameters θ i are computed using the updated gradient descent: DISPLAYFORM4 Once the models and related evaluations are generated for all batch tasks, the outer loop is activated. It consists in operating a meta-gradient update of the initial model parameters with a meta-step size β on the basis of the previous level rewards R Ti (θ i): DISPLAYFORM5 The steps detailed above are iterated until an accepted performance is reached. The ing model initialization θ * should be able to achieve fast driving adaptation after only a few gradient steps. In this section we evaluate the performance of the continuous-adapting MRL model on the challenging task of urban autonomous driving. The goal of our experiment is to demonstrate the effectiveness of meta-level learning combined with a NN controller to optimize the RL policy and achieve a more robust learning of high-dimensional and complex environments. At this stage of work, we present the preliminary of our study assessing 2 basic assumptions. The MRL agent is adapting faster at training time and displaying better generalization capabilities in unseen environments. Environment settings. We conduct our experiments using CARLA simulator for autonomous driving BID5 BID21 designed as a server-client system. Carla 3D environment consists of static objects and dynamic characters. As we consider the problem of autonomous driving in changing conditions, we induce nonstationary environments across training episodes by varying several server settings. The task complexity: select one of the available towns as well as different start and end positions for the vehicle tasks (straight or with-turn driving). FORMULA1 The traffic density: control the number of dynamic objects such as pedestrians and vehicles. Weather and lightening: select a combination of weather and illumination conditions to diversify visual effects controlling sun position, radiation intensity, cloudiness and precipitation. Hence we can exclusively use a subset of environments for meta-training ("seen") and a second subset for test-time adaptation ("unseen"). The reward is shaped as a weighted sum of the distance traveled to target, speed in km/h, collisions damage and overlaps with sidewalk and opposite lane. Results. Given the preliminary level of experiments and the absence of various state-of-the-art work on the recent CARLA simulator, we adopt BID7 methodology consisting in comparing the continuous-adapting MRL initialization with conventionally pre-trained and randomly initialized RL algorithms. In all experiments the average episodic reward is used to describe the methods global performance. An episode is terminated when the target destination is reached or after a collision with a dynamic character. FIG1 depicts the test-time adaptation performance of the 3 models. During this phase, the RL agent initialized with meta-learning still uses the NN controller for continuous adaptation. The confirm that our approach generates models adapting faster in "unseen" environments comparatively to the standard RL strategies. Zooming the initial driving steps (figure 1), we notice that our method has distinctly surpassed the standard RL versions only after 10000 steps (500 gradient descents). Subsequently we should lead further tests to identify a specific threshold for few shot learning when evolving from low to high-dimensional settings like autonomous driving task. In order to evaluate the generalization assumption, we compare the models behavior on "seen" and "unseen" environments. FIG2 does not reveal a significant "shortfall" of our approach performance between the 2 scenarios reflecting its robustness in non-stationary conditions. In the contrary, the performance of the pre-trained standard RL decreased notably in "unseen" environments due to the lack of generalization capabilities. Although all indicate a certain robustness of the continuous-adapting MRL, it is too early to draw firm at this preliminary stage of evaluation. First, the episodic reward indicator should be completed with the percentage of successfully ended episodes in order to demonstrate the effective learning of the agent and allow the comparison with state-of-the-art work BID5 BID16. Second, further consideration should be addressed to the pertinence of few shot learning regimes in very complex and high dimensional environments like autonomous driving since the meta-learned strategy may acquires a particular bias at training time "that allows it to perform better from limited experience but also limits its capacity of utilizing more data" BID27. In this paper we addressed the limits of RL algorithms in solving high-dimensional and complex tasks. Built on gradient-based meta-learning, the proposed approach implements a continuous process of policy assessment and improvement using a NN controller. Evaluated on the challenging problem of autonomous driving using CARLA simulator, our approach showed higher performance and faster learning capabilities than conventionally pre-trained and randomly initialized RL algorithms. Considering this paper as a preliminary attempt to scale up RL approaches to high-dimensional real world applications like autonomous driving, we plan in future work to bring deeper focus on several sides of the approach such as the reward function, CNN architecture and including vehicle characteristics in the tasks complexity setup.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A meta-reinforcement learning approach embedding a neural network controller applied to autonomous driving with Carla simulator.

We identify and formalize the memorization problem in meta-learning and solve this problem with novel meta-regularization method, which greatly expand the domain that meta-learning can be  applicable to and effective on.


Question: Represent the scientific article to find a short TLDR.

Question: The information bottleneck principle is an elegant and useful approach to representation learning. In this paper, we investigate the problem of representation learning in the context of reinforcement learning using the information bottleneck framework, aiming at improving the sample efficiency of the learning algorithms. We analytically derive the optimal conditional distribution of the representation, and provide a variational lower bound. Then, we maximize this lower bound with the Stein variational (SV) gradient method. We incorporate this framework in the advantageous actor critic algorithm (A2C) and the proximal policy optimization algorithm (PPO). Our experimental show that our framework can improve the sample efficiency of vanilla A2C and PPO significantly. Finally, we study the information-bottleneck (IB) perspective in deep RL with the algorithm called mutual information neural estimation(MINE). We experimentally verify that the information extraction-compression process also exists in deep RL and our framework is capable of accelerating this process. We also analyze the relationship between MINE and our method, through this relationship, we theoretically derive an algorithm to optimize our IB framework without constructing the lower bound. In training a reinforcement learning algorithm, an agent interacts with the environment, explores the (possibly unknown) state space, and learns a policy from the exploration sample data. In many cases, such samples are quite expensive to obtain (e.g., requires interactions with the physical environment). Hence, improving the sample efficiency of the learning algorithm is a key problem in RL and has been studied extensively in the literature. Popular techniques include experience reuse/replay, which leads to powerful off-policy algorithms (e.g., (; ; ; a;)), and model-based algorithms (e.g., ). Moreover, it is known that effective representations can greatly reduce the sample complexity in RL. This can be seen from the following motivating example: In the environment of a classical Atari game: Seaquest, it may take dozens of millions samples to converge to an optimal policy when the input states are raw images (more than 28,000 dimensions), while it takes less samples when the inputs are 128-dimension pre-defined RAM data . Clearly, the RAM data contain much less redundant information irrelevant to the learning process than the raw images. Thus, we argue that an efficient representation is extremely crucial to the sample efficiency. In this paper, we try to improve the sample efficiency in RL from the perspective of representation learning using the celebrated information bottleneck framework . In standard deep learning, the experiments in show that during the training process, the neural network first "remembers" the inputs by increasing the mutual information between the inputs and the representation variables, then compresses the inputs to efficient representation related to the learning task by discarding redundant information from inputs (decreasing the mutual information between inputs and representation variables). We call this phenomena "information extraction-compression process" "information extraction-compression process" "information extraction-compression process"(information E-C process). Our experiments shows that, similar to the shown in , we first (to the best of our knowledge) observe the information extraction-compression phenomena in the context of deep RL (we need to use MINE for estimating the mutual information). This observation motivates us to adopt the information bottleneck (IB) framework in reinforcement learning, in order to accelerate the extraction-compression process. The IB framework is intended to explicitly enforce RL agents to learn an efficient representation, hence improving the sample efficiency, by discarding irrelevant information from raw input data. Our technical contributions can be summarized as follows: 1. We observe that the "information extraction-compression process" also exists in the context of deep RL (using MINE to estimate the mutual information). 2. We derive the optimization problem of our information bottleneck framework in RL. In order to solve the optimization problem, we construct a lower bound and use the Stein variational gradient method developed in to optimize the lower bound. 3. We show that our framework can accelerate the information extraction-compression process. Our experimental also show that combining actor-critic algorithms (such as A2C, PPO) with our framework is more sample-efficient than their original versions. 4. We analyze the relationship between our framework and MINE, through this relationship, we theoretically derive an algorithm to optimize our IB framework without constructing the lower bound. Finally, we note that our IB method is orthogonal to other methods for improving the sample efficiency, and it is an interesting future work to incorporate it in other off-policy and model-based algorithms. Information bottleneck framework was first introduced in . They solve the framework by iterative Blahut Arimoto algorithm, which is infeasible to apply to deep neural networks. tries to open the black box of deep learning from the perspective of information bottleneck, though the method they use to compute the mutual information is not precise. derives a variational information bottleneck framework, yet apart from adding prior target distribution of the representation distribution P (Z|X), they also assume that P (Z|X) itself must be a Gaussian distribution, which limits the capabilities of the representation function. extends this framework to variational discriminator bottleneck to improve GANs , imitation learning and inverse RL. As for improving sample-efficiency, (; ; a) mainly utilize the experience-reuse. Besides experience-reuse, tries to learn a deterministic policy, seeks to mitigate the delay of off-policy. learn the environment model. Some other powerful techniques can be found in . State representation learning has been studied extensively, readers can find some classic works in the overview . Apart from this overview, (b) shows a theoretical foundation of maintaining the optimality of representation space. proposes a new perspective on representation learning in RL based on geometric properties of the space of value function. learns representation via information bottleneck(IB) in imitation/apprenticeship learning. To the best of our knowledge, there is no work that intends to directly use IB in basic RL algorithms. A Markov decision process(MDP) is a tuple, (X, A, R, P, µ), where X is the set of states, A is the set of actions, R: X × A × X → R is the reward function, P: is the transition probability function(where P (X ′ |X, a) is the probability of transitioning to state X ′ given that the previous state is X and the agent took action a in X), and µ: X → is the starting state distribution. A policy π: X → P(A) is a map from states to probability distributions over actions, with π(a|X) denoting the probability of choosing action a in state X. In reinforcement learning, we aim to select a policy π which maximizes is the expected return by policy π after taking action a in state X. Actor-critic algorithms take the advantage of both policy gradient methods and valuefunction-based methods such as the well-known A2C . Specifically, in the case that policy π(a|X; θ) is parameterized by θ, A2C uses the following equation to approximate the real policy gradient where R t = ∑ ∞ i=0 γ i r t+i is the accumulated return from time step t, H(p) is the entropy of distribution p and b(X t) is a baseline function, which is commonly replaced by V π (X t). A2C also includes the minimization of the mean square error between R t and value function V π (X t). Thus in practice, the total objective function in A2C can be written as: where α 1, α 2 are two coefficients. In the context of representation learning in RL, The information bottleneck framework is an information theoretical framework for extracting relevant information, or yielding a representation, that an input X ∈ X contains about an output Y ∈ Y. An optimal representation of X would capture the relevant factors and compress X by diminishing the irrelevant parts which do not contribute to the prediction of Y. In a Markovian structure X → Z → Y where X is the input, Z is representation of X and Y is the label of X, IB seeks an embedding distribution P ⋆ (Z|X) such that: Under review as a conference paper at ICLR 2020 for every X ∈ X, which appears as the standard cross-entropy loss 1 in supervised learning with a MI-regularizer, β is a coefficient that controls the magnitude of the regularizer. Next we derive an information bottleneck framework in reinforcement learning. Just like the label Y in the context of supervised learning as showed in, we assume the supervising signal Y in RL to be the accurate value R t of a specific state X t for a fixed policy π, which can be approximated by an n-step bootstrapping function be the following distribution:.This assumption is heuristic but reasonable: If we have an input X t and its relative label Y t = R t, we now have X t's representation Z t, naturally we want to train our decision function V π (Z t) to approximate the true label Y t. If we set our target distribution to be For simplicity, we just write P (R|Z) instead of P (Y t |Z t) in the following context. With this assumption, equation can be written as: The first term looks familiar with classic mean squared error in supervisd learning. In a network with representation parameter ϕ and policy-value parameter θ, policy lossĴ(Z; θ) in equation and IB loss in can be jointly written as: where I(X, Z; ϕ) denotes the MI between X and Z ∼ P ϕ (·|X). Notice that J(Z; θ) itself is a standard loss function in RL as showed in. Finally we get the ultimate formalization of IB framework in reinforcement learning: The following theorem shows that if the mutual information I(X, Z) of our framework and common RL framework are close, then our framework is near-optimality. Theorem Theorem Theorem 1 (Near-optimality theorem). Policy π r = π θ r, parameter ϕ r, optimal policy π ⋆ = π θ ⋆ and its relevant representation parameter ϕ ⋆ are defined as following: Define J Assume that for any In this section we first derive the target distribution in and then seek to optimize it by constructing a variational lower bound. We would like to solve the optimization problem in: Combining the derivative of L 1 and L 2 and setting their summation to 0, we can get that We provide a rigorous derivation of in the appendix(A.2). We note that though our derivation is over the representation space instead of the whole network parameter space, the optimization problem and the ing distribution are quite similar to the one studied in in the context of Bayesian inference. However, we stress that our formulation follows from the information bottleneck framework, and is mathematically different from that in . In particular, the difference lies in the term L 2, which depends on the the distribution P ϕ (Z | X) we want to optimize (while in , the corresponding term is a fixed prior). The following theorem shows that the distribution in is an optimal target distribution (with respect to the IB objective L). The proof can be found in the appendix(A.3). Theorem Theorem Theorem 2. (Representation Improvement Theorem) Consider the objective function, given a fixed policy-value parameter θ, representation distribution P ϕ (Z|X) and state distribution P (X). Define a new representation distribution: Though we have derived the optimal target distribution, it is still difficult to compute P ϕ (Z). In order to resolve this problem, we construct a variational lower bound with a distribution U (Z) which is independent of ϕ. Notice that. Now, we can derive a lower bound of L(θ, ϕ) in as follows: Naturally the target distribution of maximizing the lower bound is: Next we utilize the method in to optimize the lower bound. Stein variational gradient descent(SVGD) is a non-parametric variational inference algorithm that leverages efficient deterministic dynamics to transport a set of particles to approximate given target distributions Q(Z). We choose SVGD to optimize the lower bound because of its ability to handle unnormalized target distributions such as. Briefly, SVGD iteratively updates the "particles" via a direction function Φ ⋆ (·) in the unit ball of a reproducing kernel Hilbert space (RKHS) H: where Φ * (·) is chosen as a direction to maximally decrease 2 the KL divergence between the particles' distribution P (Z) and the target distribution Q(Z) =Q In fact, Φ * is chosen to maximize the directional derivative of F (P) = −DKL(P ||Q), which appears to be the "gradient" of F distribution, C is normalized coefficient) in the sense that where P [ϵΦ] is the distribution of Z + ϵΦ(Z) and P is the distribution of Z. showed a closed form of this direction: where K is a kernel function(typically an RBF kernel function). Notice that C has been omitted. In our case, we seek to minimize, which is equivalent to maximizeL(θ, ϕ), the greedy direction yields: In practice we replace log U (Ẑ) with ζ log U (Ẑ) where ζ is a coefficient that controls the magnitude of ∇Ẑ log U (Ẑ). Notice that Φ(Z i) is the greedy direction that Z i moves towardŝ L(θ, ϕ)'s target distribution as showed in(distribution that maximizesL(θ, ϕ)). This means Φ(Z i) is the gradient ofL(Z i, θ, ϕ): Since our ultimate purpose is to update ϕ, by the chain rule, Φ(Z i) is given in equation. In practice we update the policy-value parameter θ by common policy gradient algorithm since: and update representation parameter ϕ by. This section we verify that the information E-C process exists in deep RL with MINE and our framework accelerates this process. Mutual information neural estimation(MINE) is an algorithm that can compute mutual information(MI) between two high dimensional random variables more accurately and efficiently. Specifically, for random variables X and Z, assume T to be a function of X and Z, the calculation of I(X, Z) can be transformed to the following optimization problem: The optimal function T ⋆ (X, Z) can be approximated by updating a neural network T (X, Z; η). With the aid of this powerful tool, we would like to visualize the mutual information between input state X and its relative representation Z: Every a few update steps, we sample a batch of inputs and their relevant representations and compute their MI with MINE, every time we train MINE(update η) we just shuffle {Z i} n i=1 and roughly assume the shuffled representations {Z Figure is the tensorboard graph of mutual information estimation between X and Z in Atari game Pong, x-axis is update steps and y-axis is MI estimation. More details and can be found in appendix(A.6) and (A.7). As we can see, in both A2C with our framework and common A2C, the MI first increases to encode more information from inputs("remember" the inputs), then decreases to drop irrelevant information from inputs("forget" the useless information). And clearly, our framework extracts faster and compresses faster than common A2C as showed in figure(b). (a) MI in A2C (b) MI in A2C with our framework After completing the visualization of MI with MINE, we analyze the relationship between our framework and MINE. According to , the optimal function T * in goes as follows: Combining the with Theorem, we get: Through this relationship, we theoretically derive an algorithm that can directly optimize our framework without constructing the lower bound, we put this derivation in the appendix(A.5). In the experiments we show that our framework can improve the sample efficiency of basic RL algorithms(typically A2C and PPO). Our anonymous code can be found in https://github. com/AnonymousSubmittedCode/SVIB. Other can be found in last two appendices. In A2C with our framework, we sample Z by a network ϕ(X, ϵ) where ϵ ∼ N (·; 0, 0.1) and the number of samples from each state X is 32, readers are encouraged to take more samples if the computation resources are sufficient. We set the IB coefficient as β = 0.001. We choose two prior distributions U (Z) of our framework, the first one is uniform distribution, apparently when U (Z) is the uniform distribution, ∇Ẑ log U (Ẑ) |Ẑ =Z can be omitted. The second one is a Gaussian distribution, which is defined as follows: for a given state X i, sample a batch of {Z We also set ζ as 0.005∥∇Ẑ 1 β J(Ẑ; θ)/∇Ẑ log U (Ẑ)∥ |Ẑ =Z to control the magnitude of ∇Ẑ log U (Ẑ) |Ẑ =Z. Following , the kernel function in we used is the Gaussian RBF kernel K(Z i, Z j) = exp(−∥Z i − Z j ∥ 2 /h) where h = med 2 /2 log(n + 1), med denotes the median of pairwise distances between the particles {Z . As for the hyper-parameters in RL, we simply choose the default parameters in A2C of Openaibaselines(https://github.com/openai/baselines/tree/master/baselines/a2c). In summary, we implement the following four algorithms: A2C with uniform SVIB A2C with uniform SVIB A2C with uniform SVIB: Use ϕ(X, ϵ) as the embedding function, optimize by our framework(algorithm(A.4)) with U (Z) being uniform distribution. A2C with Gaussian SVIB A2C with Gaussian SVIB A2C with Gaussian SVIB: Use ϕ(X, ϵ) as the embedding function, optimize by our framework(algorithm(A.4)) with U (Z) being Gaussian distribution. A2C A2C A2C:Regular A2C in Openai-baselines with ϕ(X) as the embedding function. A2C with noise A2C with noise A2C with noise(For fairness):A2C with the same embedding function ϕ(X, ϵ) as A2C with our framework. Figure(a)-(e) show the performance of four A2C-based algorithms in 5 gym Atari games. We can see that A2C with our framework is more sample-efficient than both A2C and A2C with noise in nearly all 5 games. Figure 2: (a)-(e) show the performance of four A2C-based algorithms, x-axis is time steps(2000 update steps for each time step) and y-axis is the average reward over 10 episodes, (f)-(h) show the performance of four PPO-based algorithms, x-axis is time steps(300 update steps for each time step). We make exponential moving average of each game to smooth the curve(In PPO-Pong, we add 21 to all four curves in order to make exponential moving average). We can see that our framework improves sample efficiency of basic A2C and PPO. Notice that in SpaceInvaders, A2C with Gaussian SVIB is worse. We suspect that this is because the agent excessively drops information from inputs that it misses some information related to the learning process. There is a more detailed experimental discussion about this phenomena in appendix(A.7). We also implement four PPO-based algorithms whose experimental settings are same as A2C except that we set the number of samples as 26 for the sake of computation efficiency. Results can be found in the in figure(f)-(h). We study the information bottleneck principle in RL: We propose an optimization problem for learning the representation in RL based on the information-bottleneck framework and derive the optimal form of the target distribution. We construct a lower bound and utilize Stein Variational gradient method to optimize it. Finally, we verify that the information extraction and compression process also exists in deep RL, and our framework can accelerate this process. We also theoretically derive an algorithm based on MINE that can directly optimize our framework and we plan to study it experimentally in the future work. According to the assumption, naturally we have: Notice that if we use our IB framework in value-based algorithm, then the objective function J π can be defined as: where and d π is the discounted future state distribution, readers can find detailed definition of d π in the appendix of . We can get: We show the rigorous derivation of the target distribution in. Denote P as the distribution of X, P Z ϕ (Z) = P ϕ (Z) as the distribution of Z. We use P ϕ as the short hand notation for the conditional distribution P ϕ (Z|X). Moreover, we write Take the functional derivative with respect to P ϕ of the first term L 1: Hence, we can see that Then we consider the second term. By the chain rule of functional derivative, we have that Combining the derivative of L 1 and L 2 and setting their summation to 0, we can get that A.3 Proof of Theorem 2 (X, Z; ϕ), given a fixed policy-value parameter θ, representation distribution P ϕ (Z|X) and state distribution P (X), define a new representation distribution: Proof Proof Proof. Define I(X) as: According to the positivity of the KL-divergence, we have L(θ,φ) ≥ L(θ, ϕ). Algorithm 1 Information-bottleneck-based state abstraction in RL θ, ϕ ← initialize network parameters β, ζ ← initialize hyper-parameters in ϵ ← learning rate M ← number of samples from A.5 Integrate MINE to our framework MINE can also be applied to the problem of minimizing the MI between Z and X where Z is generated by a neural network P ϕ (·|X): A.6 Study the information-bottleneck perspective in RL Now we introduce the experimental settings of MI visualization. And we show that the agent in RL usually tends to follow the information E-C process. We compare the MI(I(X, Z)) between A2C and A2C with our framework. Every 2000 update steps(2560 frames each step), we re-initialize the parameter η, then sample a batch of inputs and their relevant representations, n = 64, and compute the MI with MINE. The learning rate of updating η is same as openai-baselines' A2C: 0.0007, training steps is 256 and the network architecture can be found in our code file "policy.py". Figure is the MI visualization in game Qbert. Note that there is a certain degree of fluctuations in the curve. This is because that unlike supervised learning, the distribution of datasets and learning signals R π (X) keep changing in reinforcement learning: R π (X) changes with policy π and when π gets better, the agent might find new states, in this case, I(X, Z) might increase again because the agent needs to encode information from new states in order to learn a better policy. Yet finally, the MI always tends to decrease. Thus we can say that the agent in RL usually tends to follow the information E-C process. (a) MI in A2C (b) MI in A2C with our framework Figure 3: Mutual information visualization in Qbert. As policy π gets better, the agent might find new states, in this case, I(X, Z) might increase again because the agent needs to encode information from new states in order to learn a better policy. Yet finally, the MI always tends to decrease. Thus it follows the information E-C process. We argue that it's unnecessary to compute I(Z, Y) like : According to, if the training loss continually decreases in supervised learning(Reward continually increases as showed in figure(a) in reinforcement learning), I(Z, Y) must increase gradually. We also add some additional experimental of MI visualization in the appendix(A.7). This section we add some additional experimental about our framework. Notice that in game MsPacman, performance of A2C with our framework is worse than regular A2C. According to the MI visualization of MsPacman in figure(b), we suspect that this is because A2C with our framework drops the information from inputs so excessively that it misses some information relative to the learning process. To see it accurately, in figure(b), the orange curve, which denotes A2C with our framework, from step(x-axis) 80 to 100, suddenly drops plenty of information. Meanwhile, in figure(b), from step(xaxis) 80 to 100, the rewards of orange curve start to decrease. As showed in figure, unlike Pong, Breakout, Qbert and some other shooting games, the frame of MsPacman contains much more information related to the reward: The walls, the ghosts and the tiny beans everywhere. Thus if the agent drops information too fast, it may hurt the performance.

Represent the short summary for finding the corresponding scientific article (long)

Derive an information bottleneck framework in reinforcement learning and some simple relevant theories and tools.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.