Question: Represent the scientific article to find a short TLDR

Question: Determinantal point processes (DPPs) is an effective tool to deliver diversity on multiple machine learning and computer vision tasks. Under deep learning framework, DPP is typically optimized via approximation, which is not straightforward and has some conflict with diversity requirement. We note, however, there has been no deep learning paradigms to optimize DPP directly since it involves matrix inversion which may in highly computational instability. This fact greatly hinders the wide use of DPP on some specific objectives where DPP serves as a term to measure the feature diversity. In this paper, we devise a simple but effective algorithm to address this issue to optimize DPP term directly expressed with L-ensemble in spectral domain over gram matrix, which is more flexible than learning on parametric kernels. By further taking into account some geometric constraints, our algorithm seeks to generate valid sub-gradients of DPP term in case when the DPP gram matrix is not invertible (no gradients exist in this case). In this sense, our algorithm can be easily incorporated with multiple deep learning tasks. Experiments show the effectiveness of our algorithm, indicating promising performance for practical learning problems. Diversity is desired in multiple machine learning and computer vision tasks (e.g., image hashing (; Carreira-Perpinán &), descriptor learning, metric learning and video summarization ), in which sub-sampled points or learned features need to spread out through a specific bounded space. Originated from quantum physics, determinantal point processes (DPP) have shown its power in delivering such properties b). Compared with other diversity-oriented techniques (e.g., entropy and orthogonality ), DPP shows its superiority as it incorporates only one single metric and delivers genuine diversity on any bounded space; ). Therefore, DPP has been utilized in a large body of diversity-oriented tasks. In general, sample points from a DPP tend to distribute diversely within a bounded space A. Given a positive semi-definite kernel function κ: A × A → R, the probability of a discrete point set X ⊂ A under a DPP with kernel function κ can be characterized as: where L is a |X | × |X | matrix with entry L ij = κ(x i, x j) and x i, x j ∈ X. L is called L-ensemble. Note that A is a continuous space, whereas X is finite. In the Hilbert space associated with κ, larger determinant implies larger spanned volume, thus the mapped points tend not to be similar or linearly dependent. DPP can be viewed from two perspectives: sampling and learning. A comprehensive introduction to mathematical fundamentals of DPP for sampling from a discrete space can be found in. Based on this, a line of works has been proposed (a; ;). In this paper, we concentrate on learning DPPs. In learning of DPP, the term det(L) is typically treated as a singleton diversity measurement and is extended to learning paradigms on continuous space (; ;). There are generally two lines of strategies to learn DPPs: Approximation. This type of methods is to convert DPP into a simpler format which can ease and stabilize the computation. low-rank approximation proves powerful in easing the computational burden , in which the gram matrix is factorized as L = BB where B ∈ n×m with m n. This decomposition can also reduce the complexity which is originally a cubic time of |L|. Kulesza & Taskar (2011b) explicitly expressed the kernel with κ(x, y) = σ 1 σ 2 δ(x) δ(y), where σ measures the intrinsic quality of the feature and δ(·) is function mapping input x to a feature space. In this sense, the pairwise similarity is calculated in Euclidean feature space with cosine distance. suggest approximating a given distribution by approximating the eigenvalues of the corresponding DPP. As such, the computation can be eased and become stable. Following this, DPP is also applied on some visual tasks, such as video summarization , ranking and image classification . It can be noted that the approximation is not straightforward for DPP, thus cannot fully deliver the diversity property (e.g. ing in rank-deficiency). Direct optimization. While the aforementioned methods optimize DPP with specific approximation, a series of efforts also seek to optimize the DPP term directly (; ;). In this setting, the whole gram matrix L corresponding to the pairwise similarity among features is updated directly, which allows accommodating more flexible feature mapping functions rather than an approximation. proposed an Expectation-Maximization algorithm to update marginal kernel DPP K = L(L + I) −1, together with a baseline K-Ascent derived from projected gradient ascent . extended DPP from a fixed-point perspective and proposed to optimize DPP upon a lower bound in variational inference fashion. A key problem of such line of works is that the computation is not differentiable, making it difficult to be used in deep learning frameworks. To the best of our knowledge, there is no previous method incorporating DPP as a feature-level diversity metric in deep learning. A key difficulty in doing so is that the calculation of the gradient of det(L) involves matrix inversion, which can be unstable and inaccurate in GPUs. Though KAscent seems to be a naive rule, it still needs explicit matrix inversion in the first step before the projection procedure. This fact greatly hinders the tight integration of DPP with deep networks. Some alternative methods seek to reach diversity under more constrained settings. For example, resorted to a global pairwise orthogonality constraint in hyper-sphere and employed statistical moments to measure the diversity. However, compared with DPP, such measurements are unable to fully characterize diversity in an arbitrary bounded space. In this paper, rather than providing more efficient DPP solvers, we concentrate on delivering a feasible feature-level DPP integration under the deep learning framework. To this end, we revisit the spectral decomposition of DPP and propose a sub-gradient generation method which can be tightly integrated with deep learning. Our method differs from either approximation or direct optimization by introducing a "differentiable direct optimization" procedure, thus can produce genuinely diverse features in continuous bounded space. Our method is stable and scalable to the relatively large dataset with a specific mini-batch sampling strategy, which is verified by several experiments on various tasks. Notations: Bold lower case x and bold upper case K represent vector and matrix, respectively. det(·) and Tr(·) calculate the determinant and trace of a matrix, respectively. A ⊗ B is the element-wise product of matrices A and B. |X | and |x| measure the cardinality of a finite set X and the L 2 length of a vector x, respectively. x, y calculates the inner product of the two vectors. x = diag(X) transforms a diagonal matrix X into its vector form x, and vice versa. We refer "positive semi-definite" and "positive definite" to PSD and PD, respectively. Denote the real numbers. 2.1 DETERMINANTAL POINT PROCESS L-ensemble expression of DPP requires L to be PSD, whereas kernel expression further constrains K < I (each eigenvalue of K is less than 1). A conversion from L to K can thus be written as, which is the marginal normalization constant given a specific L. While there is always conversion from L to K, the inverse may not exist. In practice, one may construct L-ensemble first, then normalize it into a marginal kernel. This fact may give rise to the difficulty of deep networks. Since a conversion from K to L might not exist, the network needs carefully adjusting the gradients under specific constraints to ensure the updated L to be valid. As L and K share the same eigenvectors v i, a pair of L and K holds the relation: where λ i is the ith eigenvalue. It is seen that such conversion is not straightforward to be directly integrated with deep learning framework. Therefore, we optimize ensemble L directly in this paper. We briefly introduce Gaussian kernel in this section, which works on Hilbert space with infinite dimension. Mercer's theorem ensures the PSD properties when constructing new kernels with existing ones under a specific procedure. Such procedure is also employed in multiple kernel learning paradigms (; b;), which is out of the scope of this paper. A Gaussian kernel is defined as κ(where σ is a controlling parameter. Thus an L-ensemble matrix becomes According to the definition, L ii = 1 and for any element in the matrix we have L ij ∈. With Gaussian kernel, we have a nice property 0 ≤ det(L) ≤ 1. This can be easily verified by applying geometric inequality to the eigenvalues of L. Although not tight, this property shows that the determinant value with Gaussian kernel is bounded. This fact inspires one version of our algorithm detailed in the next section. Throughout this paper, our discussion is based on the Gaussian kernel unless specified. Given vectorized inputs I i ∈ R h where i = 1,..., n, our goal is to learn a map f such that the features x i = f (I i) can spread out within a bounded feature space x i ∈ S. Hereafter we refer space to an Euclidean bounded space (e.g., [−1, 1] d ) without loss of generality. Given any loss function J, the chain rule of gradient involving DPP is written as: where X refer to the features before DPP layer. While calculating ∂J/∂ det(L) and ∂L/∂X is straightforward, the main difficulty lies on the calculation of ∂ det(L)/∂L. We will discuss the calculation of this term under two case: 1) When the inversion L −1 can be stably obained, we will derive the gradient of DPP det(L) on Sec 3.1; When L is not invertible or L −1 is difficult to calculate, we give the procedure to handle the case by generating valid sub-gradient in Sec 3.2. Since our objective is to diverse features, det(L) will serve as a (partial) objective term to be directly maximized. With kernel κ, a DPP regularization term seeks to maximize the possibility of a feature configuration x i, i = 1,..., n. As this possibility is proportional to det(L), the objective is max det(L). This can become a regularization term where diversity is required. Thus with a general loss function L G, our aim is to solve min L G − λ 1 det(L), with the controlling parameter λ 1 ≥ 0. For the time being, we assume that kernel matrix L is invertible (we will discuss the case when L is not invertible in the next section), hence L −1 exists. Without loss of generality, we discuss the gradient of the determinant equipped with Gaussian kernel. For other kernels the derivation is analogous. According to Eq (??), L ij can be further factorized as: where x ij is the jth dimension of feature x i. Using chain rule, the derivative of det(L) w.r.t. x il can be written as: where on the ijth position of the corresponding element is: Eq can be more compactly expressed as: where M (il) is such a matrix that, except for the ith column and row, all resting elements are 0s. Besides, the ijth and jith elements of. In summary, Eq can be simplified as: To ease the computation and fully utilize the chain rule in deep learning architecture, we peel the DPP loss into two layers, and the corresponding gradient product can be expressed as: While we can use existing package to obtain ∂L/∂x reliably, the way to stably calculate ∂ det(L)/∂L becomes essential. We will detail in the next section once the term is hard to calculate. The calculation of the gradient ∂ det L/∂L involves computing the inverse matrix L −1. However, the kernel matrix L is not always invertible. This situation happens iff there exists at least a pair of features x i and x j such that x i = x j. In this case, there exist two identical columns/rows of L and the 0 eigenvalue in the non-invertibility. This phenomenon is sometimes caused by Relu function, which can map different input values onto an identical one. Even when all features are distinct, the numerical precision (typically on float number in GPU) may also lead to failure. We occasionally observed that GPU calculation of L −1 reports error even no eigenvalue is 0. One may imagine a naive replacement of matrix inverse with the pseudo-inverse, which can be applied on singular matrices. However, pseudo-inverse will keep the zero eigenvalues intact (still rank-deficiency), and the back-propagated gradient will play no part to increase the determinant value (both 0 before and after updates). To address this, we first diverge to consider the objective of DPP max det(L). Since DPP term seeks to maximize the determinant, for a configuration can be a valid ascending direction. Thus we give the following definition: We see if a proper sub-gradientL can be found at det(L) = 0, back-propagation procedure in deep learning can consequently perform calculation usingL. To obtain suchL, we first note that L can be eigen-decomposed as following since it is symmetric and PSD: where U is the orthogonal eigenvector matrix and Λ's diagonal elements are the corresponding eigenvalues. As L has zero eigenvalues, the rank of Λ is lower than the dimension of L. We sort all eigenvalue into descending order to k = (σ 1, ..., σ q, 0, ..., 0), where q < n. We then employ a simple yet effective amplification procedure by amplifying any eigenvalue smaller than ∆ to ∆. The amplified eigenvalues are nowk = (σ 1, ..., σ s, ∆, ..., ∆), where s ≤ q. Let the diagonalized amplified eigenvalue matrix beΛ (w.r.t. k), then the modified matrix with small positive determinant can be written as:L For any > 0, we can choose a sufficiently small ∆ such that det(L) <. Thus the continuity of this procedure is guaranteed. The differenceL =L − L can be viewed as a proper ascending direction w.r.t. L, as by addingL, det(L +L) becomes above 0 as well as arbitrarily small. It is trivial to prove thatL is a sub-gradient on a neighbor of L, thusL is also a proper sub-gradient sufficing Definition 3.1. This procedure is summarized in Algorithm 1 and is termed as DPPSG. Intuitively, once encountering an identical or too close feature pair x i and x j, this procedure tries to enhance the diversity by separating them apart from each other. Inspired by geometric inequality, we provide an improved version of the algorithm taking into account the property of Gaussian kernel. First it easy to show that the function i σ i is concave in the feasible set i σ i = n (diagonal of Gaussian gram matrices are 1s, thus trace is n) and the maximal objective is reached out iff σ i = 1. Therefore, any point b = (1 − θ)(σ 1, ..., σ n) + θ(1, ..., 1) will increase the objective i σ i. By letting θ being a small value, the proper sub-gradient becomes Udiag(b − σ)U, where σ = (σ 1, ..., σ n). This version of update differs from DPPSG as it generates sub-gradients under geometric constraints. The method is summarized in Algorithm 2 and is termed as DPPSG*. During implementation, the irregularity of L is examined to determine whether to adopt a normal back-propagation (in Sec 3.1) or sub-gradient (in Sec 3.2). This can be done by verifying if the determinant value in the forward pass is less than a pre-defined small value β. This proper subgradient based back-propagation method can be used to integrate to deep learning framework with other objectives involving matrix determinant. We emphasize that our method is different from the line of gradient-projection based methods, such as K-Ascent. While projection-based methods calculate the true gradient then project it back to a feasible set, our methods generate proper subgradient directly. Without explicitly computing matrix inversion, sub-gradients, in this case, is more feasible for deep learning framework. We employed a balanced sampling strategy for each mini-batch. Assuming the batch size is n and there are c classes in total, in each mini-batch the distribution of samples generally follows the whole training sample distribution on c classes. This strategy is considered to utilize the intrinsic diversity of the original data. Besides, mini-batch sampling can constrain the overhead of DPP computation depending only on the batch size, which can be viewed as a constant in practice. Practically, the features are always required to lie in a bounded space. This is essential in some applications as a bounded space is more controllable. Especially, sometimes one may demand that the features should suffice to a pre-defined distribution P. This bounding requirement is crucial to the objective of DPP since maximizing determinant tends to draw feature points infinitely apart from each other. A naive method to achieve this is to truncate the features or using barrier functions. However, these methods will in irregularly dense distribution on the learned feature space boundary. To overcome this issue, we employ Wasserstein GAN (WGAN) to enforce the features mapped to a specific distribution P. As we do not focus on WGAN in this paper, readers are referred to for more details. To this end, we randomly sample n 1 pointsx i from the distribution P under balanced sampling, which are treated as positive samples. The generator f (·) takes a feature as input and outputs the corresponding embedding. Denote the discriminator h(·) (which is also the mapping from input to feature). Then the WGAN loss for discriminator is: According to the , we incorporate the generator loss In this section, we conduct two experiments. One is about metric learning and image hashing on MNIST and CIFAR to verify the effectiveness of our method, while another is for local descriptor retrieval task based on HardNet . For the first test. MNIST This simple dataset is suitable to reveal the geometric properties of the features on various tasks. We test the image retrieval task equipped with contrastive loss where L(i) indicates the label of the ith feature and x i is the learnt feature. We employ a simple network structure for MNIST. This network consists of 3 convolutional layers (Conv) followed by 2 fully connected layers (FC). Batch normalization is applied on each layer. The filter number of each Conv are 32, 32 and 64, respectively. The sizes of the filter are identically 5 × 5. For the first Conv, we employ maxpooling. For the other 2 Convs, average pooling is adopted. The dimensions of the last FCs are 200 and 2 (for 2D visualization). The performance can be found in Table 1 and the feature distribution is visualized in Figure 1. From Table 1, it is observed that the performance on retrieval task can be enhanced by adding the DPP and WGAN regularization terms. We see that DPP term can enhance the retrieval performance by avoiding feature points from concentrating too much. In this sense, the learned map around the separating boundary can be much smoother. As retrieval task typically requires the existence of top-k inter-class samples rather than concentrating property, the DPP term is more preferable. In Figure 1 (c), we see that the feature points generally fall into the pre-defined space [−1, 1] 2. The utility of such space is high without sacrificing the retrieval performance. Typically, DPPSG* is slightly superior to DPPSG. Thus in the following test, we only report the performance under DPPSG* setting (termed as DPP* for short). We conduct image hashing on CIFAR-10 which seeks to produce binary code for images. To this end, we follow the binary hashing code generation procedure in which is activated by a Sigmoid function. The number of neurons in the second last layer equals to the number of bits of the hashing codes. It is anticipated that DPP regularization can enhance the utility in binary code space since the code can spread out 2. We test two lengths of binary code (12 and 16). We visualize the 16-bit feature distribution using TSNE in Fig. 2 (a) and (b), and the binary code histogram comparison in (c). The quantitative are summarized in Table 2. jointly solve binary code generation and classification, we report both retrieval performance (mAP) and classification performance (Acc). We see our method can significantly enhance the binary space utility while keeping the performance almost intact. We employ all the convolutional layers in VGG-19 as the base and discard its final fully connected layers. Thus the output size of this base VGG-19 network is 1 × 1 × 512. We concatenate 3 fully connected layers with ReLU activation on each after that with dimensions 512, 100 and 20, respectively. Contrastive loss is applied on the 20-dimensional space. We train the whole network from scratch. Aside from mAP, we also report top-k average precision (Precision-k) and the Wasserstein distance to the pre-defined distribution (Gap to P). The performance on coarse (20 classes) and fine (100 classes) levels can be found in Table 3. In either setting, we see that DPP+WGAN significantly outperform the baseline. Thus we infer that the DPP term can serve as a regularization not only for the feature itself but also for the smoothness of the mapping. Since the DPP term avoids the features from concentrating too much, the learned mapping should also be from a smoother function family. Batch size VS. performance We study how batch size influences the performance with DPP regularization. To this end, we report the performance on CIFAR-100 100-class retrieval with different batch sizes. The are shown in Table 2. Generally, with larger batch size, the algorithm can reach out better mAP. We note the computational efficiency of DPP sub-gradients is high, which adds very slight overhead (even with 500 batch size) to each iteration of common back-propagation under contrastive loss, which can be neglected. Precision-k (%) Gap to P k 10 20 50 10 20 50 On coarse Table 4: Image hashing on CIFAR-10. "Acc" is the classification accuracy. This test utilizes the UBC Phototour dataset , which consists of three subsets (Liberty, Notre Dame, and Yosemite) with around 400k 64 × 64 local patches for each. We follow the protocol in to treat two subsets as the training set and the third one as the testing set. As each pair of matched image patches includes only two patches, there is no need to apply balanced sampling in this test. We simply add DPP regularization term to the objective of state-of-the-art algorithm HardNet . The batch size is 512. We report FPR (false positive rate) and FDR (false discovery rate) following;. Results are summarized in Table 5. Several baselines are selected for comparison (i.e. SIFT , MatchNet , TFeat-M , L2Net and HardNet ). As the authors improved HardNet after the NeurIPS submission, we also compare with the latest version (termed as HardNet+). We only conduct our method under DPPSG* setting and name our method HardDPP. We see that with DPP regularization, the performance of HardNet can be further enhanced. Note that in HardNet there is no WGAN integrated as the mapped features lie in the surface of a hyper unit sphere. While the sampling strategy of HardNet emphasizes the embedding behavior near the margin, DPP regularization can further focus on global feature distribution. Table 5: Performance of UBC Phototour comparison. Notre, Yose and Lib are short for "Notre Dame", "Yosemite" and "Liberty", respectively. , we report FPR at true positive rate at 95%. The best are in bold. In this paper, we investigated the problem of learning diverse features via a determinantal point process under deep learning framework. To overcome the instability in computing the gradient which involves the matrix inverse, we developed an efficient and reliable procedure called proper spectral sub-gradient generation. The generated proper sub-gradient can replace the true gradient and performs well in applications. We also considered how to constrain the features into a bounded space, since in such a way one can ensure the behavior of the network more predictable. To this end, we further incorporated Wasserstein GAN into our framework. Together, DPP+WGAN showed significant performance on both some common criteria and feature space utility. A APPENDIX MNIST Some parameters are set as follows: α = 5, λ 1 = 10 3, λ 2 = 10 6, margin µ = 0.8, variance for Gaussian kernel σ = 0.2 and ∆ = 10 −7. During the training, the batch size is set to 200. In each iteration of DPP and WGAN training, we uniformly sample 2, 000 adversarial points from the space [−1, 1] 2. We adopt RMSprop and the learning rate is 10 −4 for all tests. In the testing stage, we sample 2, 000 points from [−1, 1] 2 and calculate the Wasserstein distance with all the testing samples. This procedure is conducted 10 times and the mean distance is reported. CIFAR-10 image hashing The parameters in the hashing related experiments are used as following: variance for Gaussian Kernel σ = 2, the coefficients for the loss term of DPP is λ 1 = 10 2 and for the loss term of discriminator and generator in WGAN is 10 and 1 respectively. The batch size is set to 500 and the learning rate is initialized to 0.01 with a changing rate of 0.1 at every 150 epoch. The total number of epoch is set to 350 and we adopt the Adam optimizer to update our model. The parameter setting is as follows: α = 1, λ 1 = 10 3, λ 2 = 10 3, margin µ = 0.8, variance for Gaussian kernel σ = 0.2 and ∆ = 10 −6. The rest of the settings are the same as those of MNIST test. A.2 CRITERIA PRECISION-k AND MAP-k For image retrieval task, we adopt the top-k mean average precision (abbreviated as mAP-k) to evaluate the performance. We also present the top-k average precision (abbreviated as Precision-k), which is calculated as: where b is the corresponding class and I is the indicator function: Thus mAP-k is the reweighted version of Precision-k: A.3 OVERHEAD OF DPP Calculating SVD or matrix inversion on a large number of features can be time consuming. However, in our setting, we employed a common practice in deep learning -mini-batch -to avoid such computation on a whole batch. We can conclude that mini-batch strategy can limit the computational cost such that the extra overhead of DPP is only dependent on the batch size (thus other parts of the networks have no impact on this overhead). Therefore, although the complexity of our method is O(n 3), n only corresponds to the batch size rather than whole sample number in our setting, which is much more manageable in practice. We report the average overhead comparison on CIFAR-10 hashing task with varying batch sizes (100, 200, 250, 400 and 500) on a GTX 1080 GPU as in Table 6 Table 6: Overhead of a single batch and a DPP calculation on CIFAR-10 hashing task with varying batch size. Time is in seconds. where "overhead-all" and "overhead-DPP" refer to the average time cost (s) for a single batch on all the computation and only DPP computation (both forward and backward), respectively. We can conclude that, compared to other computation, the extra overhead of DPP is small (even in a simple network as CIFAR-10 hashing). Besides, a batch size up to 500 is considered to be sufficient in most of the applications. In practice, we did not employ any trick to reduce such overhead (since it is out of the papers focus) but simply utilized standard functions provided by Pytorch. For MNIST verification test, we employed a simple backbone. The structure of the backbone is {conv 1(5 × 5)+maxpool+conv 2(5 × 5)+avepool+conv 3(5 × 5)+avepool+fully con1(200-d)+fully con2(2-d)+fully con3(10-d)+contrastive loss}. We add DPP and WGAN regularization to the features at "fully con2" layer, which is 2-dimensional thus better for visualization. For CIFAR-10 image hashing task, we employ the same network structure as a high-cited method DCH . DPP and WGAN loss is applied on the second last fully connected layer (the dimension of this layer corresponds to the length of digits in the hashing code). For CIFAR-100 metric learning task, we employ all the convolutional layers of VGG19, concatenated with 3 fully connected layers (with 512, 100 and 20 dimension). DPP and WGAN loss, together with contrastive loss, is applied on the final fully connected layer (20-dimension). The network is trained from scratch without any pre-training. A.5 DEGRADATION ON CIFAR-10 IMAGE HASHING For the performance degradation with DPP on hashing task, we can take Figure 2 (c) as an example to explain. We see that original DCH features concentrate on several digits (generally 10 digits corresponding to 10 classes), while DPP features diffuse to almost the whole discrete space. In this sense, if one retrieves the k-th closest hashing code, DCH can find the hashing code with a small searching radius. However, one has to greatly enlarge the search radius for k-th closest code in DPP feature space since the distribution is much more even. In this sense, DPP will inevitably causes degradation since large searching radius will more likely to reach a code in other class. Therefore, we think "utility vs mAP" is an intrinsic conflict and needs to reach a trade-off.

Represent the short summary for finding the corresponding scientific article (long)

We proposed a specific back-propagation method via proper spectral sub-gradient to integrate determinantal point process to deep learning framework.

represent the short summary for finding the corresponding scientific article (long)

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.


Question: Represent the article for finding a single-sentence TLDR

Question: The quality of a machine translation system depends largely on the availability of sizable parallel corpora. For the recently popular Neural Machine Translation (NMT) framework, data sparsity problem can become even more severe. With large amount of tunable parameters, the NMT model may overfit to the existing language pairs while failing to understand the general diversity in language. In this paper, we advocate to broadcast every sentence pair as two groups of similar sentences to incorporate more diversity in language expressions, which we name as parallel cluster. Then we define a more general cluster-to-cluster correspondence score and train our model to maximize this score. Since direct maximization is difficult, we derive its lower-bound as our surrogate objective, which is found to generalize point-point Maximum Likelihood Estimation (MLE) and point-to-cluster Reward Augmented Maximum Likelihood (RAML) algorithms as special cases. Based on this novel objective function, we delineate four potential systems to realize our cluster-to-cluster framework and test their performances in three recognized translation tasks, each task with forward and reverse translation directions. In each of the six experiments, our proposed four parallel systems have consistently proved to outperform the MLE baseline, RL (Reinforcement Learning) and RAML systems significantly. Finally, we have performed case study to empirically analyze the strength of the cluster-to-cluster NMT framework. Recently, an encode-decoder neural architecture has surged and gained its popularity in machine translation. In this framework, the encoder builds up a representation of the source sentence and the decoder uses its previous RNN hidden state and attention mechanism to generate target translation. In order to better memorize the input information, an attention mechanism has been exploited to further boost its performance. In order to train the attentive encoder-decoder architecture, Maximum Likelihood Estimation (MLE) algorithm has been widely used, which aims at maximizing the point-to-point (one sentence to one sentence) log-likelihood of data pairs in a given dataset. However, this algorithm has severely suffered from data sparsity problem, or in other word, maximizing only likelihood the existing language pairs might make the model blind to all the non-existing similar sentence pairs. Thus, the large neural model might overfit to certain prototypes existing in the training set while failing to generalize more unseen but similar scenarios in test time.hurting its semantic meaning. 2) Model-Centroid Augmentation (RL), and BID13 leverage model-generated candidates as pseudo training samples, which are weighted with rewards to enhance the model learning. By exploring self-generated candidates, the model is able to understand the diversity in the output space. In pseudo-learning algorithms, both RAML and RL can be interpreted as broadcasting a target ground truth as a cluster of analogues while leaving the source input untouched, which though helps the model understand target diversity, fails to capture the input diversity. In order to explore both sides' diversity, we advocate a novel and general cluster-to-cluster framework of pseudo learning, which first broadcasts both source and target sentence as clusters and then train the model to comprehend their correspondence, as described in FIG0.In this paper, we first introduce the concept of parallel cluster, then design the cluster-to-cluster correspondence score as our optimization objective, based on which, we derive its lower bound KL-divergence as our surrogate objective for model training. In order to realize our proposed framework, we design four parallel systems and apply them to three recognized machine translation tasks with both forward and reverse translation directions, these four systems have all demonstrated their advantages over the existing competing algorithms in six translation tasks. In the appendices, we draw samples from the parallel clusters and further analyze their properties to verify our motivation. The contributions of our paper can be summarized as follows: 1) We are the first to propose the concept of cluster-to-cluster framework, which provides a novel perspective to current sequence-tosequence learning problems. 2) We delineate the framework and arrive in a novel KL-divergence loss function and generalizes several existing algorithms as special cases, which provides a highlevel understanding about the previous algorithms.2 RELATED LITERATURE Exposure bias and train-test loss discrepancy are two major issues in the training of sequence prediction models. Many research works BID16 BID13 BID9 have attempted to tackle these issues by adding reward-weighted samples drawn from model distribution into the training data via a Reinforcement Learning BID17 framework. By exposing the model to its own distribution, these methods are reported to achieve significant improvements., BID13 and BID16 advocate to optimize the sequence model as a stochastic policy to maximize its expected task-level reward. Though RL is not initially designed to resolve data sparsity problem, the model-centroid training samples can indeed alleviate data sparseness by exposing the sequence-to-sequence model to more unseen scenarios. One problem of the previous RL works is that, the input information is still restricted to the dataset, which fails to teach model to comprehend source diversity. The cluster-to-cluster framework augments many similar input sentences to account for source language diversity. One successful approach for data augmentation in neural machine translation system is Reward Augmented Maximum Likelihood (RAML), which proposes a novel payoff distribution to augment training samples based on task-level reward (BLEU, Edit Distance, etc). In order to sample from this intractable distribution, they further stratify the sampling process as first sampling an edit distance, then performing random substitution/deletion operations. Following the work of RAML, BID11 introduces a novel softmax Q-Distribution to reveal RAML's relation with Bayes decision rule, and they also propose an alternative sampling strategy -first randomly replacing n-gram of the ground truth sentence and then using payoff distribution to compute corresponding importance weight with local normalization. These two approaches augment the target-side data by exposing the model to diverse scenarios and improve its robustness. We draw our inspiration from RAML, but with a difference that, instead of based on task-level reward, a learnable payoff function (cluster distribution) is used in our approach to take more latent structures into account, such as semantic meaning, language fluency, etc. From the cluster distribution, we can sample semantically and syntactically correct candidates to train the model. In addition, our more generalized bilateral data augmentation strategy also empowers our model more capability to generalize better. In order to utilize the large amount of monolingual data in current NMT framework, different strategies have been designed, the most common methods can be concluded into these categories: 1) using large monolingual data to train language model and integrates it to enhance language fluency BID2. 2) using self-learning method to transform the monolingual data into bilingual form BID14 ). 3) using reconstruction strategy to leverage monolingual data to enhance NMT training BID3. Although our motivation to augment training data is aligned with these semi-supervised algorithms, our proposed framework has substantial differences from them: 1) we don't rely on additional monolingual data to boost NMT performance; 2) Though we jointly train forward and backward translation models as advocated in and BID3, our joint algorithm doesn't involve any interactions between these two models (they can be trained independently). We define the parallel cluster as two groups of weighted sentences C(Y *) and C(X *), whose similarities (BLEU, METEOR, etc) with Y * and X * are above certain threshold M. DISPLAYFORM0 Every sample X or Y is associated with a normalized weight p(X|X *) or p(Y |Y *) to denote how much chance a sentence X or Y is sampled from the corresponding cluster, here we draw a schematic diagram to better visualize the parallel cluster in FIG0. We will further talk about how we define and compute the weights in the following sections. Upon the definition of parallel cluster, we further design a cluster-to-cluster correspondence score CR c→c (X *, Y *) as the log scaled expectation of likelihood of a random sentence X in source cluster C(X *) being translated to Y in target cluster C(Y *), which generally denotes the translatability of two clusters, formally, we define the cluster-to-cluster correspondence score CR c→c (X *, Y *) as below: The higher correspondence score the more likely these two clusters correspond to each other. Note that the cluster-to-cluster correspondence score can reflect both NMT's and cluster's quality, assuming the cluster is ideal, then the correspondence score measures the translatability from a source sentence X to a target sentence Y, while assuming the NMT is ideal, then the correspondence score measures the quality of the cluster (the capability to rank paraphrases based on semantically similarity). DISPLAYFORM0 DISPLAYFORM1 Based on the definition of parallel cluster and cluster-to-cluster correspondence score, we further design the cluster-to-cluster framework's objective function as maximizing the empirical correspondence score CR c→c (X *, Y * ; D) with the regularization of target cluster's entropy H(p(Y |Y *)) in a dataset D, as described below: DISPLAYFORM0 By applying Jensen's inequality to the objective function Obj c→c, we can further derive its lowerbound as: DISPLAYFORM1 From this, we notice that the cluster-to-cluster objective is lower bounded by a negative KL- DISPLAYFORM2. Therefore, we can use this lower-bound to maximize correspondence score, by changing the sign of this lower-bound function, we further define the loss function as: DISPLAYFORM3 We theoretically verify that this lower bound KL-divergence can generalize Maximum Likelihood (MLE) and Reward Augmented Maximum Likelihood (RAML) as special cases when we instantiate cluster distribution as Kronecker-Delta function δ(Y |Y *) and payoff dis- TAB0. DISPLAYFORM4 In this section, we try to minimize the proposed KL-divergence KL(p(Y |Y *)||p(Y |X *)) so as to raise the lower bound of the regularized cluster-to-cluster correspondence. We can write its deriva-tives w.r.t to the NMT parameters in two forms, namely parallel sampling and NMT broadcasting modes, which differ in their Monte-Carlo proposing distribution.• Parallel Sampling: sampling candidates independently from two clusters and then reweighted pairwise samples with a translation confidence w(Y |X, X *). DISPLAYFORM0 • Translation Broadcasting: sampling candidates from one cluster and broadcasting them through the NMT to construct its opponents, and re-weighted by cluster confidence c(Y |Y *, X *). DISPLAYFORM1 More specifically, translation broadcasting's samples are more NMT-aware in the sense that it incorporates NMT's knowledge to generate correspondents. The parallel sampling mode works like twosided RAML while translation broadcasting works more like mixed RAML-RL . In this paper, we design cluster distribution in two manners, namely inadaptive (pre-computed without training) and adaptive (trained during optimization) cluster. Both cluster designs meet the criterion of concentrating around the ground truth according to sentence similarity metric. In addition, a cutoff criterion is also leveraged to reject samples whose task-level score is lower than certain threshold M value as in Equation 1.• Inadaptive Cluster: we use two non-parametric distributions q(X|X *) and q(Y |Y *) to denote source and target parallel clusters, based on the similarity score between sample X/Y and the ground truth X * /Y *. We follow the payoff distribution to define our inadaptive cluster: DISPLAYFORM0 where R(Y, Y *) denotes the task-level reward (BLEU, CIDEr, METEOR, etc) and R(Y, Y *) denotes its normalization in the whole output space, τ is the hyper-parameter temperature to control the smoothness of the optimal distribution around correct target Y *. Since the task-level reward only considers string-level matching (precision, recall, etc) while ignoring semantic coherence, the generated samples though lexically similar, prone to many semantical and syntactical mistakes, which might cause counter-effects to the NMT model.• Adaptive Cluster: we use two parametric models p(X|X *) and p(Y |Y *) to denote the source and target adaptive cluster, which follow encoder-decoder neural architecture but take ground truth X *, Y * as inputs. Adaptive cluster is designed to fulfill the following two requirements: 1) Proximity to ground truth: the randomly sampled candidates should have high similarity with the ground truth. 2) High correspondence score: parallel cluster should be highly correlated and translatable. Combining these two goals can guarantee mutual dependence between the source and target clusters and also retain its similarity to the original ground truth. Formally, we write the optimization target of the target cluster as: DISPLAYFORM1 During optimization, we fix the forward NMT p(Y |X) and target cluster p(X|X *) to update source cluster p(Y |Y *), and we fix the parameters of backward NMT p(X|Y) and source cluster p(Y |Y *) to update target cluster p(X|X *). Here we write target cluster's derivative as following: DISPLAYFORM2 Due to the mutual dependence between adaptive clusters and translation models, we advocate to alternately update the cluster and the translation models. In this section, we advocate to combine both forward and backward translation directions in a joint system to simultaneously learn four models -forward NMT p(Y |X), backward NMT (X|Y), source cluster p(X|X *) and target cluster p(Y |Y *). We exploit different scenarios to combine these four models and then design four parallel systems, whose implementations are elaborated in TAB2. System-A and B use inadaptive (non-parametric) cluster, thus require optimizing only the two translation systems; system-A applies parallel sampling algorithm while B applies translation broadcasting algorithm. In contrast, system-C and D apply adaptive (parametric) cluster, thus require simultaneous optimization of both NMT and cluster, system-C applies parallel sampling while system-D applies translation broadcasting algorithm. These four systems exhibit different characteristics which are shown in details as below: In a slight abuse of notation, we will denote DISPLAYFORM0 System-A For system-A, we use inadaptive cluster with parallel sampling strategy to train the NMT model, and the forward-backward joint objective functions is defined as: DISPLAYFORM1 Formally, the derivative respect to η and γ are shown as: DISPLAYFORM2 Parallel candidates are sampled from source and target cluster distributions are leveraged by scaled translation scores w(X|Y, Y *), w(Y |X, X *) during optimization. System-B With the same loss function in system-A, translation broadcasting is leveraged to compute derivatives in system-B, instead of parallel sampling, and the gradients is shown as: DISPLAYFORM3 This system works similar as reinforcement Learning, where normalized environmental rewards R(X, X *),R(Y, Y *) are leveraged to guide the model's policy search, and the gradients is interpreted as a form of Monte-Carlo Policy Gradient BID18.System-C Unlike System-A and system-B, two adaptive cluster distributions is used in system-C, thus the NMT and cluster are jointly optimized during training, and the loss function is defined as: DISPLAYFORM4 we can get the derivatives as below: DISPLAYFORM5 To train the NMT system, parallel sentence pairs (X, Y) are firstly sampled from two independent cluster distributions and then translation confidence scores w(Y |X, X *), w(X|Y, Y *) are leveraged to guide the training. The derivatives w.r.t the cluster contain two elements, candidates sampled from translation system, and candidates sampled from cluster itself. The two components together ensure parallel cluster's translatability and the similarity to the ground truth. System-D With the same loss function in system-C, translation broadcasting strategy is leveraged to compute derivatives, instead of parallel sampling, and the gradients is shown as: DISPLAYFORM6 System-D works quite similar as system-B but differs in that cluster confidence scores c(X|X *, Y *), c(Y |Y *, X *) are leveraged in training NMT, hence it is more abundant than tasklevel rewards (R(X, X *) andR(Y, Y *)). System-D adopts the same gradient formulas in system-C to update the clusters. The details of the training algorithm for system-A,B,C,D are shown in Algorithm 1: To evaluate our cluster-to-cluster NMT framework on different-sized (small-data, medium-data and large-data) and different-lingual (German-English and Chinese-English) translation tasks, we conduct experiments on three datasets (IWSLT, LDC, WMT). For more details about the datasets, please refer to Appendix C. For comparability, we follow the existing papers to adopt similar network architectures, and apply learning rate annealing strategy described in to further boost our baseline NMT system. In our experiments, we design both the NMT and adaptive cluster models based on one-layered encoder-decoder network ) with a maximum sentence length of 62 for both the encoder and decoder. During training, ADADELTA is adopted with = 10 −6 and ρ = 0.95 to separately optimize the NMT's and adaptive cluster's parameters. During decoding, a beam size of 8 is used to approximate the full search space. We compute the threshold similarity M via sentence-BLEU, some small-scaled experiments indicate M = 0.5 yields best performance, so we simply stick to this setting throughout all the experiments. To prevent too much hyper-parameter tuning in building the inadaptive cluster, we follow to select the best temperature τ = 0.8 in all experiments. For comparison, RAML and RL systems are also implemented with the same sequence-to-sequence attention model, following and BID18. For more details of our RL's and RAML's implementations, please refer to Appendix A. We can see from TAB5 that our system-D achieves significant improvements on both directions. Though our baseline system is already extremely strong, using cluster-to-cluster framework can further boost the NMT system by over 1.0 BLEU point. Baseline Model Baseline Model MIXER BID13 20.10 21.81 --BSO BID19 24.03 26.36 --A-C 27.56 28.53 --Softmax-Q BID11 27.66 28.77 --Our implementation of RL BID18 29.10 29.70 24.40 24.75 Our implementation of RAML Table 4: Experimental on NIST Chinese-English Machine Translation Task WMT2014 German-English We can see from Table 5 that system-C achieves the strongest on both WMT14 EN-DE and DE-EN tasks, which outperforms the baseline system by over 1.1 BLEU points. It's worth noting that our one-layer RNN model even outperforms the deep multilayer RNN model of Zhou et al. FORMULA0 and, which contain a stack of 4-7 LSTM layers. By using cluster-to-cluster framework, our one-layer RNN model can fully exploit the dataset and learn to generalize better. From the above 24 parallel cluster-to-cluster experiments, we observe general improvements over the fine-tuned baseline systems as well as our implemented RL/RAML systems. To understand the strength of our cluster-to-cluster framework, we give more detailed comparisons with existing competing algorithms as below:Comparison with RAML From the above three tables, we can observe general improvements yielded by RAML algorithm on different tasks (except LDC Chinese-English), but RAML still suffers from two problems: on one hand, RAML's benefits is restricted by its neglect of the input variabilities, and on the other hand, without considering semantic contexts and language fluency, Table 5: Experimental on WMT-2014 German-English Machine Translation Task RAML's random replacement strategy may introduce noisy and wrong bilingual pairs to hurt the translation performance (like in LDC Chinese-English translation task). Our adaptive cluster takes into account more semantic contexts to enclose more rational paraphrases, and the bilateral augmentation also empowers the model more chance to access various inputs. Comparison with RL We can also observe prevalent improvements yielded by RL algorithm BID13. Exposing the model to self-generated translation can improve the performance. Our methods inherit this merit and further enhance it with source and target clusters, which can improve the model with more sampled bilingual pairs from both source and target sides. Comparison between four parallel systems Among our proposed four parallel systems, system-C and D achieve better performances than A and B throughout different experiments, which confirms the advantages of the adaptive clusters. The adaptive cluster is more flexible and target optimized than inadaptive cluster. Unlike the payoff distribution used in inadaptive cluster which only takes task-level reward into account, the adaptive cluster learns more sophisticated criterion and thus assigns more rational probability to sampled candidates. We give more detailed analysis and visualization in the appendices to demonstrate how the source and target clusters look like. We demonstrate the learning curves of four systems and visualize some adaptive clusters in Appendix D and Appendix E, which give a more intuition about cluster-to-cluster learning. In this paper, we propose a cluster-to-cluster learning framework and incorporate this concept into neural machine translation. Our designed systems have proved to be efficient in helping current NMT model to generalize in both source and target sides. In the cluster-to-cluster framework, the cooperation of four agents can augment valuable samples and alleviate data sparsity, and achieve significant improvement compared with strong baseline systems. We believe the concept of clusterto-cluster learning can be applicable to a wide range of natural language or computer vision tasks, which will be explored in the future. Appendices A SYSTEM-DESIGN Sequence to sequence problem (machine translation) can be considered to produce an output sequence Y = (y 1, y 2, . . ., y T), y t ∈ A given an input X. Given input-target pairs (X, Y *), the generated sequence Y on test is evaluated with task-specific score R(Y, Y *). Recurrent neural networks have been widely used in sequence to sequence prediction tasks. As proposed in and, the basic idea is to first encode the input sequence as a variablelength feature vectors, then apply attention mechanism to compute weighted average over the input vectors and summarize a context vector, with which, previous hidden states and previous label are fed into the decoder RNN to predict the next state and its label. In our approach, attention-based encoder-decoder is leveraged for both the translation and cluster models, shown as: DISPLAYFORM0 A.1 RL NMT In order to train our RL system as well as adaptive cluster, we need to define a task-level reward as driving signal. Instead of directly applying BLEU or other evaluation metric, we advocate to use a surrogate n-gram match interpolation, as shown as: DISPLAYFORM1 where N n denotes the number of n-gram match between Y and Y *. In order to alleviate sequencereward sparseness, we further split it as a series of local reward to drive model's policy search at every time step. Formally, we write the step-wise reward r(y t |y 1:t−1, Y *) as following. where N (Y,Ỹ) represents the occurrence of n-gramỸ in sequence Y, specifically, if a certain nsequence y t−n+1:t appears in reference and it's not repeating more than needed, then we assign a corresponding matching score to y t, the policy gradient is described as: DISPLAYFORM2 DISPLAYFORM3 A.2 RAML NMT In order to sample from the intractable payoff distribution for system-A/B as well as our implemented RAML system, we adopt stratified sampling technique described in. Given a sentence Y *, we first sample an edit distance m, and then randomly select m positions to replace the original labels. For each sentence, we randomly sample four candidates to perform RAML training. DISPLAYFORM4 B MATHEMATICAL ANALYSIS We optimize the model parameters of our cluster-to-cluster models by minimizing the lower-bound KL-divergence instead of maximizing the original correspondence score, to characterize the difference between the two objective function, we analyze the relationships between these two functions below: DISPLAYFORM5 which can be further written as: DISPLAYFORM6 therefore, we can derive: DISPLAYFORM7 Since both cluster and translation confidence score c(Y |Y *, X *) and w(Y |X, X *) require computing the marginalized probability p(Y |X *) known to be intractable for variable-length sequences, here we adopt different mechanisms to approximate them. In system-A and C, we simplify DISPLAYFORM8 pη(Y |X *). In system-B and D, since Y is broadcast through the translation system, the marginalized probabilityp(Y |X *) is close to one, we discard this factor and approximate c(Y |Y DISPLAYFORM9 IWSLT2014 Dataset The IWSLT2014 German-English training data set contains 153K sentences while the validation data set contains 6,969 sentences pairs. The test set comprises dev2010, dev2012, tst2010, tst2011 and tst2012, and the total amount are 6,750 sentences. We adopt 512 as the length of RNN hidden stats and 256 as embedding size. We use bidirectional encoder and initialize both its own decoder states and coach's hidden state with the learner's last hidden state. The experimental for IWSLT2014 German-English and English-German Translation Task are summarized in TAB5. In order to give a more intuitive view about what the cluster distribution looks like, we draw some samples from the well-trained cluster distribution in LDC Chinese-English Translation task as shown in TAB9. we can observe that most of the paraphrases are based on three types of modification, namely form changing, synonym replacement as well as simplification. Most of the modifications does not alter the original meaning of the ground truth. Encompassing more expressions with close meanings can ease the data sparseness problem, and enhance its generalization ability. We here draw two samples from source and target clusters in FIG3, which demonstrates how point-point correspondence can be expanded into cluster-to-cluster correspondence. Reference taihsi natives seeking work in other parts of the country are given a thorough UNK before being hired, and later their colleagues maintain a healthy distance at first.Cluster taihsi natives seeking work in other parts of the country are given a thorough UNK before being employed, and later their colleagues maintain a healthy distance at first. Property Simplification Reference i once took mr tung chee -hwa to a squatter area where he found beyond imagination that a narrow alley could have accommodated so many people. Cluster i once took mr tung chee -hwa to a squatter area where he fo und beyond imagination that a narrow alley have a lot of people.

represent the short summary for finding the corresponding scientific article (long)

We invent a novel cluster-to-cluster framework for NMT training, which can better understand the both source and target language diversity.

Represent the short summary for finding the corresponding scientific article (long)

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The capability of making interpretable and self-explanatory decisions is essential for developing responsible machine learning systems. In this work, we study the learning to explain the problem in the scope of inductive logic programming (ILP). We propose Neural Logic Inductive Learning (NLIL), an efficient differentiable ILP framework that learns first-order logic rules that can explain the patterns in the data. In experiments, compared with the state-of-the-art models, we find NLIL is able to search for rules that are x10 times longer while remaining x3 times faster. We also show that NLIL can scale to large image datasets, i.e. Visual Genome, with 1M entities.: A scene-graph can describe the relations of objects in an image. The NLIL can utilize this graph and explain the presence of objects Car and Person by learning the first-order logic rules that characterize the common sub-patterns in the graph. The explanation is globally consistent and can be interpreted as commonsense knowledge. The recent years have witnessed the growing success of deep learning models in a wide range of applications. However, these models are also criticized for the lack of interpretability in its behavior and decision making process , and for being data-hungry. The ability to explain its decision is essential for developing a responsible and robust decision system . On the other hand, logic programming methods, in the form of first-order logic (FOL), are capable of discovering and representing knowledge in explicit symbolic structure that can be understood and examined by human . In this paper, we investigate the learning to explain problem in the scope of inductive logic programming (ILP) which seeks to learn first-order logic rules that explain the data. Traditional ILP methods (Galárraga et al., 2015) rely on hard matching and discrete logic for rule search which is not tolerant for ambiguous and noisy data . A number of works are proposed for developing differentiable ILP models that combine the strength of neural and logicbased computation (; ; Rocktäschel & ; ;). Methods such as ∂ILP are referred to as forward-chaining methods. It constructs rules using a set of pre-defined templates and evaluates them by applying the rule on data multiple times to deduce new facts that lie in the held-out set (related works available at Appendix A). However, general ILP problem involves several steps that are NP-hard: (i) the rule search space grows exponentially in the length of the rule; (ii) assigning the logic variables to be shared by predicates grows exponentially in the number of arguments, which we refer as variable binding problem; (iii) the number of rule instantiations needed for formula evaluation grows exponentially in the size of data. To alleviate these complexities, most works have limited the search length to within 3 and resort to template-based variable assignments, limiting the expressiveness of the learned rules (detailed discussion available at Appendix B). Still, most of the works are limited in small scale problems with less than 10 relations and 1K entities. On the other hand, multi-hop reasoning methods (; ; ; ;) are proposed for the knowledge base (KB) completion task. Methods such as NeuralLP can answer the KB queries by searching for a relational path that leads from the subject to the object. These methods can be interpreted in the ILP domain where the learned relational path is equivalent to a chain-like first-order rule. Compared to the template-based counterparts, methods such as NeuralLP is highly efficient in variable binding and rule evaluation. However, they are limited in two aspects: (i) the chain-like rules represent a subset of the Horn clauses, and are limited in expressing complex rules such as those shown in Figure 1; (ii) the relational path is generated while conditioning on the specific query, meaning that the learned rule is only valid for the current query. This makes it difficult to learn rules that are globally consistent in the KB, which is an important aspect of a good explanation. In this work, we propose Neural Logic Inductive Learning (NLIL), a differentiable ILP method that extends the multi-hop reasoning framework for general ILP problem. NLIL is highly efficient and expressive. We propose a divide-and-conquer strategy and decompose the search space into 3 subspaces in a hierarchy, where each of them can be searched efficiently using attentions. This enables us to search for x10 times longer rules while remaining x3 times faster than the state-of-theart methods. We maintain the global consistency of rules by splitting the training into rule generation and rule evaluation phase, where the former is only conditioned on the predicate type that is shared globally. And more importantly, we show that a scalable ILP method is widely applicable for model explanations in supervised learning scenario. We apply NLIL on Visual Genome dataset for learning explanations for 150 object classes over 1M entities. We demonstrate that the learned rules, while maintaining the interpretability, have comparable predictive power as densely supervised models, and generalize well with less than 1% of the data. Supervised learning typically involves learning classifiers that map an object from its input space to a score between 0 and 1. How can one explain the outcome of a classifier? Recent works on interpretability focus on generating heatmaps or attention that self-explains a classifier (; ;). We argue that a more effective and humanintelligent explanation is through the description of the connection with other classifiers. For example, consider an object detector with classifiers Person(X), Car(X), Clothing(X) and Inside(X, X) that detects if certain region contains a person, a car, a clothing or is inside another region, respectively. To explain why a person is present, one can leverage its connection with other attributes, such as "X is a person if it's inside a car and wearing clothing", as shown in Figure 1. This intuition draws a close connection to a longstanding problem of first-order logic literature, i.e. Inductive Logic Programming (ILP). A typical first-order logic system consists of 3 components: entity, predicate and formula. Entities are objects x ∈ X. For example, for a given image, a certain region is an entity x, and the set of all possible regions is X. Predicates are functions that map entities to 0 or 1, for example Person: x → {0, 1}, x ∈ X. Classifiers can be seen as soft predicates. Predicates can take multiple arguments, e.g. Inside is a predicate with 2 inputs. The number of arguments is referred to as the arity. Atom is a predicate symbol applied to a logic variable, e.g. Person(X) and Inside(X, X). A logic variable such as X can be instantiated into any object in X. A first-order logic (FOL) formula is a combination of atoms using logical operations {∧, ∨, ¬} which correspond to logic and, or and not respectively. given a set of predicates P = {P 1 ...P K}, we define the explanation of a predicate P k as a first-order logic entailment where P k (X, X) is the head of the entailment, and it will become P k (X) if it is a unary predicate. A is defined as the rule body and is a general formula, e.g. conjunction normal form (CNF), that is made of atoms with predicate symbols from P and logic variables that are either head variables X, X or one of the body variables Y = {Y 1, Y 2, ...}. By using the logic variables, the explanation becomes transferrable as it represents the "lifted" knowledge that does not depend on the specific data. It can be easily interpreted. For example, represents the knowledge that "if an object is inside the car with clothing on it, then it's a person". To evaluate a formula on the actual data, one grounds the formula by instantiating all the variables into objects. For example, in Figure 1, Eq. is applied to the specific regions of an image. Given a relational knowledge base (KB) that consists of a set of facts where P i ∈ P and x i, x i ∈ X. The task of learning FOL rules in the form of Eq. that entail target predicate P * ∈ P is called inductive logic programming. For simplicity, we consider unary and binary predicates for the following contents, but this definition can be extended to predicates with higher arity as well. The ILP problem is closely related to the multi-hop reasoning task on the knowledge graph (; ; ; ;). Similar to ILP, the task operates on a KB that consists of a set of predicates P. Here the facts are stored with respect to the predicate P k which is represented as a binary matrix M k in {0, 1} |X |×|X |. This is an adjacency matrix, meaning that x i, P k, x j is in the KB if and only if the (i, j) entry of M k is 1. Given a query q = x, P *, x. The task is to find a relational path x − −− → x, such that the two query entities are connected. Formally, let v x be the one-hot encoding of object x with dimension of |X |. Then, the (t)th hop of the reasoning along the path is represented as where M (t) is the adjacency matrix of the predicate used in (t)th hop. The v (t) is the path features vector, where the jth element v (t) j counts the number of unique paths from x to x j . After T steps of reasoning, the score of the query is computed as For each q, the goal is to (i) find an appropriate T and (ii) for each t ∈ [1, 2, ..., T], find the appropriate M (t) to multiply, such that Eq. is maximized. These two discrete picks can be relaxed as learning the weighted sum of scores from all possible paths, and weighted sum of matrices at each step. Let be the soft path selection function parameterized by (i) the path attention vector s ψ = [s ψ] T that softly picks the best path with length between 1 to T that answers the query, and (ii) the operator attention vectors S ϕ = [s ϕ softly picks the M (t) at (t)th step. Here we omit the dependence on M k for notation clarity. These two attentions are generated with a model with learnable parameters w. For methods such as , T(x; w) is a random walk sampler which generates one-hot vectors that simulate the random walk on the graph starting from x. And in NeuralLP, T(x; w) is an RNN controller that generates a sequence of normalized attention vectors with v x as the initial input. Therefore, the objective is defined as arg max Learning the relational path in the multi-hop reasoning can be interpreted as solving an ILP problem with chain-like FOL rules ) Compared to the template-based ILP methods such as ∂ILP, this class of methods is efficient in rule exploration and evaluation. However, (P1) generating explanations for supervised models puts a high demand on the rule expressiveness. The chain-like rule space is limited in its expressive power because it represents a constrained subspace of the Horn clauses rule space. For example, Eq. is a Horn clause and is not chain-like. And the ability to efficiently search beyond the chain-like rule space is still lacking in these methods. On the other hand, (P2) the attention generator T(x; w) is dependent on x, the subject of a specific query q, meaning that the explanation generated for target P * can vary from query to query. This makes it difficult to learn FOL rules that are globally consistent in the KB. In this section, we show the connection between the multi-hop reasoning methods with the general logic entailment defined in Eq.. Then we propose a hierarchical rule space to solve (P1), i.e. we extend the chain-like space for efficient learning of more expressive rules. In Eq., variables that only appear in the body are under existential quantifier. We can turn Eq. into Skolem normal form by replacing all variables under existential quantifier with functions with respect to X and X, If the functions are known, Eq. will be much easier to evaluate than Eq.. Because grounding this formula only requires to instantiate the head variables, and the rest of the body variables are then determined by the deterministic functions. Functions in Eq. can be arbitrary. But what are the functions that one can utilize? We propose to adopt the notions in section 2.2 and treat each predicate as an operator, such that we have a subspace of the functions Φ = {ϕ 1, ..., ϕ K}, where where U and B are the sets of unary and binary predicates respectively. The operator of the unary predicate takes no input and is parameterized with a diagonal matrix. Intuitively, given a subject entity x, ϕ k returns the set embedding that represents the object entities that, together with the subject, satisfy the predicate P k. For example, let v x be the one-hot encoding of an object in the image, then ϕ Inside (v x) returns the objects that spatially contain the input box. For unary predicate such as Car(X), its operator ϕ Car = M car 1 takes no input and returns the set of all objects labelled as car. Since we only use Φ, a subspace of the functions, the existential variables that can be represented by the operator calls, denoted asŶ, also form the subsetŶ ⊆ Y. This is slightly constrained from Eq.. For example, in Person(X) ← Car(Y), Y can not be interpreted as the operator call from X. However, we argue that such rules are generally trivial. For example, it's not likely to infer "an image contains a person" by simply checking if "there is any car in the image". Therefore, any FOL formula that complies with Eq. can now be converted into the operator form and vice versa. For example, Eq. can be written as where the variable Y 1 and Y 2 are eliminated. Note that this conversion is not unique. For example, Car(ϕ Inside (X)) can be also written as Inside(X, ϕ Car ). The variable binding problem now becomes equivalent to the path-finding problem in section 2.2, where one searches for the appropriate chain of operator calls that can represent the variable inŶ. Tree-like Figure 2: Factor graphs of example chainlike, tree-like and conjunctions of rules. Each rule type is the subset of the latter. Succ stands for successor. As discussed above, the Eq. is equivalent to a chain-like rule. We want to extend this notion and be able to represent more expressive rules. To do this, we introduce the notion of primitive statement ψ. Note that an atom is defined as a predicate symbol applied to specific logic variables. Similarly, we define a predicate symbol applied to the head variables or those inŶ as a primitive statement. For example, in Eq., ψ 1 = Car(ϕ Inside (X)) and ψ 2 = On(ϕ Clothing , X) are two primitive statements. Similar to an atom, each primitive statement is a mapping from the input space to a scalar confidence score, i.e. ψ:, their mappings are defined as where σ(·) is the sigmoid function. Note that we give unary ψ a dummy input x for notation convenience. For example, in Its value is computed as Compared to Eq., Eq. replaces the target v x into another relational path. This makes it possible to represent "correlations" between two variables, and the path that starts from the unary operator, e.g. ϕ Eye . To see this, one can view a FOL rule as a factor graph with logic variables as the nodes and predicates as the potentials. And running the operator call is essentially conducting the belief propagation over the graph in a fixed direction. As shown in Figure 2, primitive statement is capable of representing the tree-like factor graphs, which significantly improves the expressive power of the learned rules. Similarly, Eq. can be relaxed into weighted sums. In Eq., all relational paths are summed with a single path attention vector s ψ. We extend this notion by assigning separate vectors for each argument of the statement ψ. Let S ψ, S ψ ∈ R K×T be the path attention matrices for the first and second argument of all statements in Ψ, i.e. s ψ,k and s ψ,k are the path attention vectors of the first and second argument of the kth statement. Then we have we want to further extend the rule search space by exploring the logic combinations of primitive statements, via {∧, ∨, ¬}, as shown in Figure 2. To do this, we utilize the soft logic not and soft logic and operations where p, q ∈. Here we do not include the logic ∨ operation because it can be implicitly represented as be the set of primitive statements with all possible predicate symbols. We define the formula set at lth level as, where each element in the formula set {f : f ∈ F l} is called a formula such that f: Intuitively, we define the logic combination space in a similar way as that in pathfinding: the initial formula set contains only primitive statements Ψ, because they are formulas by themselves. For the l − 1th formula set F l−1, we concatenate it with its logic negation, which yieldŝ F l−1. Then each formula in the next level is the logic and of two formulas fromF l−1. Enumerating all possible combinations at each level is expensive, so we set up a memory limitation C to indicate the maximum number of combinations each level can keep track of 1. In other words, each level F l is to search for C logic and combinations on formulas from the previous levelF l−1, such that the cth formula at the lth level f lc is As an example, for Ψ = {ψ 1, ψ 2} and C = 2, one possible level sequence is F 0 = {ψ 1, ψ 2},..} and etc. To collect the rules from all levels, the final level L is the union of previous sets, i.e. Note that Eq. does not explicitly forbid trivial rules such as ψ 1 * (1 − ψ 1) that is always true regardless of the input. This is alleviated by introducing nonexistent queries during the training (detailed discussion at section 5). Again, the rule selection can be parameterized into the weighted-sum form with respect to the attentions. We define the formula attention tensors as S f, S f ∈ R L−1×C×2C, such that f lc is the product of two summations over the previous outputs weighted by attention vectors s f,lc and s f,lc respectively 2. Formally, we have where f l−1 (x, x) ∈ R 2C is the stacked outputs of all formulas f ∈F l−1 with arguments x, x. Finally, we want to select the best explanation and compute the score for each query. Let s o be the Iterate Concat attention vector over F L, so the output score is defined as An overview of the relaxed hierarchical rule space is illustrated in Figure 3. We have defined a hierarchical rule space as shown in Figure 3, where the discrete selections on the operators, statements and logic combinations are all relaxed into the weight sums with respect to a series of attention parameters S ϕ, S ψ, S ψ, S f, S f and s o. In this section, we solve (P2), i.e. we propose a differentiable model that generates these attentions without conditioning on the specific query. The goal of NLIL is to generate data-independent FOL rules. In other words, for each target predicate P *, its rule set F L and the final output rule should remain unchanged for all the queries q = x, P *, x (which is different from that in Eq.). To do this, we define the learnable embeddings of all predicates as H = [h 1, .., h K] T ∈ R K×d, and the embeddings for the "dummy" arguments X and X as e X, e X ∈ R d. We define the attention generation model as where h * is the embedding of P *, such that attentions only vary with respect to P *. As shown in Figure 4, we propose a stack of three Transformer networks for attention generator T. Each module is designed to mimic the actual evaluation that could happen during the operator call, primitive statement evaluation and formula computation respectively with neural networks and "dummy" embeddings. And the attention matrices generated during this simulated evaluation process are kept for evaluating Eq.. A MultiHeadAttn is a standard Transformer module such that MultiHeadAttn: where d is the latent dimension and q, v are the query and value dimensions respectively. It takes the query Q and input value V (which will be internally transformed into keys and values), and returns the output value O and attention matrix S. Intuitively, S encodes the "compatibility" between query and the value, and O represents the "outcome" of a query given its compatibility with the input. Operator search: For target predicate P *, we alter the embedding matrix H witĥ such that the rule generation is predicate-specific. Let q (t) ϕ be the learnable tth step operator query embedding. The operator transformer module is parameterized aŝ Here,V ϕ is the dummy input embedding representing the starting points of the paths. e ϕ is a learnable operator encoding such thatQ ϕ represents the embeddings of all operators Φ. Therefore, we consider thatV ϕ encodes the outputs of the operator calls of K predicates. And we aggregate the outputs with another MultiHeadAttn with respect to a single query q Primitive statement search: T be the output embedding of T paths. The path attention is generated as Here, e ψ and e ψ are the first and second argument encodings, such that Q ψ and Q ψ encode the arguments of each statement in Ψ. The compatibility between paths and the arguments are computed with two MultiHeadAttns. Finally, a FeedForward is used to aggregate the selections. Its output V ψ ∈ R K×d represents the of all statement evaluations in Ψ. Formula search: Let Q f,l, Q f,l ∈ R C×d be the learnable queries of the first and second argument of formulas at lth level, and let V f,0 = V ψ. The formula attention is generated aŝ Here, e +, e ¬ are the learnable embeddings, such thatV f,l−1 represents the positive and negative states of the formulas at l − 1th level. Similar to the statement search, the compatibility between the logic and arguments and the previous formulas are computed with two MultiHeadAttns. And the embeddings of formulas at lth level V f,l are aggregated by a FeedForward. Finally, let q o be the learnable final output query and let The training of NLIL consists of two phases: rule generation and rule evaluation. During generation, we run Eq. to obtain the attentions S ϕ, S ψ, S ψ, S f, S f and s o for all P * s. For the evaluation phase, we sample a mini-batch of queries {x,, and evaluate the formulas using Eq.. Here, y is the query label indicating if the triplet exists in the KB or not. We sample nonexistent queries to prevent the model from learning trivial rules that always output 1. In the experiments, these negative queries are sampled uniformly from the target query matrix M * where the entry is 0. Then the objective becomes arg min Since the attentions are generated from Eq. differentiably, the loss is back-propagated through the attentions into the Transformer networks for end-to-end training. During training, the from operator calls and logic combinations are averaged via attentions. For validation and testing, we evaluate the model with the explicit FOL rules extracted from the ϕ is such a distribution over random variables k ∈ [1, K]. And the weighted sum is the expectation over M k. Therefore, one can extract the explicit rules by sampling from the distributions (; . However, since we are interested in the best rules and the attentions usually become highly concentrated on one entity after convergence. We replace the sampling with the arg max, where we get the one-hot encoding of the entity with the largest probability mass. We first evaluate NLIL on classical ILP benchmarks and compare it with 4 state-of-the-art KB completion methods in terms of their accuracy and efficiency. Then we show NLIL is capable of learning FOL explanations for object classifiers on a large image dataset when scene-graphs are present. Though each scene-graph corresponds to a small KB, the total amount of the graphs makes it infeasible for all classical ILP methods. We show that NLIL can overcome it via efficient stochastic training. We evaluate NLIL together with two state-of-the-art differentiable ILP methods, i.e. NeuralLP and ∂ILP , and two structure embedding methods, TransE and RotatE . Detailed experiments setup is available at Appendix C. Benchmark datasets: (i) Even-and-Successor (ES) benchmark is introduced in , which involves two unary predicates Even(X), Zero(X) and one binary predicate Succ(X, Y). The goal is to learn FOL rules over a set of integers. The benchmark is evaluated with 10, 50 and 1K consecutive integers starting at 0; (ii) FB15K-237 is a subset of the Freebase knowledge base containing general knowledge facts; (iii) WN18 is the subset of WordNet containing relations between words. Statistics of datasets are provided in Table 2. Knowledge base completion: All models are evaluated on the KB completion task. The benchmark datasets are split into train/valid/test sets. The model is tasked to predict the probability of a fact triplet (query) being present in the KB. We use Mean Reciprocal Ranks (MRR) and Hits@10 for evaluation metrics (see Appendix C for details). Results on Even-and-Successor benchmark are shown in Table 5a. Since the benchmark is noisefree, we only show the wall clock time for completely solving the task. As we have previously mentioned, the forward-chaining method, i.e. ∂ILP scales exponentially in the number of facts and quickly becomes infeasible for 1K entities. Thus, we skip its evaluation for other benchmarks. Results on FB15K-237 and WN18 are shown in Table. 1. Compared to NeuralLP, NLIL yields slightly higher scores. This is due to the benchmarks favor symmetric/asymmetric relations or compositions of a few relations , such that most valuable rules will already lie within the chain-like search space of NeuralLP. Thus the improvements gained from a larger search space with NLIL are limited. On the other hand, with the Transformer block and smaller model created for each target predicate, NLIL can achieve a similar score at least 3 times faster. Compared to the structure embedding methods, NLIL is significantly outperformed by the current state-of-the-art, i.e. RotatE, on FB15K. This is expected because NLIL searches over the symbolic space that is highly constrained. However, the learned rules are still reasonably predictive, as its performance is comparable to that of TransE. Scalability for long rules: we demonstrate that NLIL can explore longer rules efficiently. We compare the wall clock time of NeuralLP and NLIL for performing one epoch of training against different maximum rule lengths. As shown in Figure 5b, NeuralLP searches over a chain-like rule space thus scales linearly with the length, while NLIL searches over a hierarchical space thus grows in log scale. The search time for length 32 in NLIL is similar to that for length 3 in NerualLP. The ability to perform ILP efficiently extends the applications of NLIL to beyond canonical KB completion. For example in visual object detection and relation learning, supervised models can learn to generate a scene-graph (As shown in Figure 1) for each image. It consists of nodes each labeled as an object class. And each pair of objects are connected with one type of relation. The scene-graph can then be represented as a relational KB where one can perform ILP. Learning the FOL rules on such an output of a supervised model is beneficial. As it provides an alternative way of interpreting model behaviors in terms of its relations with other classifiers that are consistent across the dataset. To show this, we conduct experiments on Visual Genome dataset . The original dataset is highly noisy , so we use a pre-processed version available as the GQA dataset . The scene-graphs are converted to a collection KBs, and its statistics are shown in Table 2. We filter out the predicates with less than 1500 occurrences. The processed KBs contain 213 predicates. Then we perform ILP on learning the explanations for the top 150 objects in the dataset. Quantitatively, we evaluate the learned rules on predicting the object class labels on a held-out set in terms of their R@1 and R@5. As none of the ILP works scale to this benchmark, we compare NLIL with two supervised baselines: (i) MLP-RCNN: a MLP classifier with RCNN features of the object (available in GQA dataset) as input; and (ii) Freq: a frequency-based baseline that predicts object label by looking at the mostly occurred object class in the relation that contains the target. This method is nontrivial. As noted in , a large number of triples in Visual Genome are highly predictive by knowing only the relation type and either one of the objects or subjects. Explaining objects with rules: Results are shown in Table 3. We see that the supervised method achieves the best scores, as it relies on highly informative visual features. On the other hand, NLIL achieves a comparable score on R@1 solely relying on KBs with sparse binary labels. We note that NLIL outperforms Freq significantly. This means the FOL rules learned by NLIL are beyond the superficial correlations exhibited by the dataset. We verify this finding by showing the rules for top objects in Table 4. Induction for few-shot learning: Logic inductive learning is data-efficient and the learned rules are highly transferrable. To see this, we vary the size of the training set and compare the R@1 scores for 3 methods. As shown in Figure 5c, the NLIL maintains a similar R@1 score with less than 1% of the training set. In this work, we propose Neural Logic Inductive Learning, a differentiable ILP framework that learns explanatory rules from data. We demonstrate that NLIL can scale to very large datasets while being able to search over complex and expressive rules. More importantly, we show that a scalable ILP method is effective in explaining decisions of supervised models, which provides an alternative perspective for inspecting the decision process of machine learning systems. Inductive Logic Programming (ILP) is the task that seeks to summarize the underlying patterns shared in the data and express it as a set of logic programs (or rule/formulae) . Traditional ILP methods such as AMIE+ (Galárraga et al., 2015) and RLvLR relies on explicit search-based method for rule mining with various pruning techniques. These works can scale up to very large knowledge bases. However, the algorithm complexity grows exponentially in the size of the variables and predicates involved. The acquired rules are often restricted to Horn clauses with a maximum length of less than 3, limiting the expressiveness of the rules. On the other hand, compared to the differentiable approach, traditional methods make use of hard matching and discrete logic for rule search, which lacks the tolerance for ambiguous and noisy data. The state-of-the-art differentiable forward-chaining methods focus on rule learning on predefined templates (; ;), typically in the form of a Horn clause with one head predicate and two body predicates with chain-like variables, i.e. To evaluate the rules, one starts with a set of facts and repeatedly apply rules for every possible triple until no new facts can be deduced. Then the deduced facts are compared with a heldout ground-truth set. Rules that are learned in this approach are in first-order, i.e. data-independent and can be readily interpreted. However, the deducing phase can quickly become infeasible with a larger set. Although ∂ILP has proposed to alleviate by performing only a fixed number of steps, works of this type could generally scale to KBs with less than 1K facts and 100 entities. On the other hand, differentiable backward-chaining methods such as NTP (Rocktäschel &) are more efficient in rule evaluation. In , NTP 2.0 can scale to larges KBs such as WordNet. However, FOL rules are searched with templates, so the expressiveness is still limited. Another differentiable ILP method, i.e. Neural Logic Machine (NLM), is proposed in , which learns to represent logic predicates with tensorized operations. NLM is capable of both deductive and inductive learning on predicates with unknown arity. However, as a forward-chaining method, it also suffers from the scalability issue as ∂ILP. It involves a permutation operation over the tensors when performing logic deductions, making it difficult to scale to real-world KBs. On the other hand, the inductive rules learned by NLM are encoded by the network parameters implicitly, so it does not support representing the rules with explicit predicate and logic variable symbols. Multi-hop reasoning: Multi-hop reasoning methods (; ; ; ; ; such as NeuralLP construct rule on-the-fly when given a specific query. It adopts a flexible ILP setting: instead of pre-defining templates, it assumes a chain-like Horn clause can be constructed to answer the query And each step of the reasoning in the chain can be efficiently represented by matrix multiplication. The ing algorithm is highly scalable compared to the forward-chaining counter-parts and can learn rules on large datasets such as FreeBase. However, this approach reasons over a single chainlike path, and the path is sampled by performing random walks that are independent on the task context , limiting the rule expressiveness. On the other hand, the FOL rule is generated while conditioning on the specific query, making it difficult to extract rules that are globally consistent. Link prediction with relational embeddings: Besides multi-hop reasoning methods, a number of works are proposed for KB completion using learnable embeddings for KB relations. For example, In (; ; Balažević et al., 2019) it learns to map KB relations into vector space and predict links with scoring functions. NTN , on the other hand, parameterizes each relation into a neural network. In this approach, embeddings are used for predicting links directly, thus its prediction cannot be interpreted as explicit FOL rules. This is different from that in NLIL, where predicate embeddings are used for generating data-independent rules. Standard ILP approaches are difficult and involve several procedures that have been proved to be NP-hard. The complexity comes from 3 levels: first, the search space for a formula is vast. The body of the entailment can be arbitrarily long and the same predicate can appear multiple times with different variables, for example, the Inside predicate in Eq. appears twice. Most ILP works constrain the logic entailment to be Horn clause, i.e. the body of the entailment is a flat conjunction over literals, and the length limited within 3 for large datasets. Second, constructing formulas also involves assigning logic variables that are shared across different predicates, which we refer to as variable binding. For example, in Eq., to express that a person is inside the car, we use X and Y to represent the region of a person and that of a car, and the same two variables are used in Inside to express their relations. Different bindings lead to different meanings. For a formula with n arguments (Eq. has 7), there are O(n n) possible assignments. Existing ILP works either resort to constructing formula from pre-defined templates or from chain-like variable reference, limiting the expressiveness of the learned rules. Finally, evaluating a formula candidate is expensive. A FOL rule is data-independent. To evaluate it, one needs to replace the variables with actual entities and compute its value. This is referred to as grounding or instantiation. Each variable used in a formula can be grounded independently, meaning a formula with n variables can be instantiated into O(C n) grounded formulas, where C is the number of total entities. For example, Eq. contains 3 logic variables: X, Y and Z. To evaluate this formula, one needs to instantiate these variables into C 3 possible combinations, and check if the rule holds or not in each case. However in many domains, such as object detection, such grounding space is vast (e.g. all possible bounding boxes of an image) making the full evaluation infeasible. Many forward-chaining methods such as ∂ILP scales exponentially in the size of the grounding space, thus are limited to small scale datasets with less than 10 predicates and 1K entities. Baselines: For NeuralLP, we use the official implementation at here. For ∂ILP, we use the thirdparty implementation at here. For TransE, we use the implementation at here. For RotatE, we use the official implementation at here. Model setting: For NLIL, we create separate Transformer blocks for each target predicate. All experiments are conducted on a machine with i7-8700K, 32G RAM and one GTX1080ti. We use the embedding size d = 32. We use 3 layers of multi-head attentions for each Transformer network. The number of attention heads are set to number of heads = 4 for encoder, and the first two layers of the decoder. The last layer of the decoder has one attention head to produce the final attention required for rule evaluation. For KB completion task, we set the number of operator calls T = 2 and formula combinations L = 0, as most of the relations in those benchmarks can be recovered by symmetric/asymmetric relations or compositions of a few relations . Thus complex formulas are not preferred. For FB15K-237, binary predicates are grouped hierarchically into domains. To avoid unnecessary search overhead, we use the most frequent 20 predicates that share the same root domain (e.g. "award", "location") with the head predicate for rule body construction, which is a similar treatment as in. For VG dataset, we set T = 3, L = 2 and C = 4. Evaluation metrics: Following the conventions in ) we use Mean Reciprocal Ranks (MRR) and Hits@10 for evaluation metrics. For each query x, P k, x, the model generates a ranking list over all possible groundings of predicate P k, with other groundtruth triplets filtered out. Then MRR is the average of the reciprocal rank of the queries in their corresponding lists, and Hits@10 is the percentage of queries that are ranked within the top 10 in the list.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An efficient differentiable ILP model that learns first-order logic rules that can explain the data.

Represent the short summary for finding the corresponding scientific article (long):

We propose TransINT, a novel and interpretable KG embedding method that isomorphically preserves the implication ordering among relations in the embedding space in an explainable, robust, and geometrically coherent way.