Question: Represent the article for finding a single-sentence TLDR

Question: Presence of bias and confounding effects is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in the recent years. Such challenges range from spurious associations of confounding variables in medical studies to the bias of race in gender or face recognition systems. One solution is to enhance datasets and organize them such that they do not reflect biases, which is a cumbersome and intensive task. The alternative is to make use of available data and build models considering these biases. Traditional statistical methods apply straightforward techniques such as residualization or stratification to precomputed features to account for confounding variables. However, these techniques are not in general applicable to end-to-end deep learning methods. In this paper, we propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s). This is enabled by incorporating a new adversarial loss function that encourages a vanished correlation between the bias and learned features. We apply our method to a synthetic, a medical diagnosis, and a gender classification (Gender Shades) dataset. Our show that the learned features by our method not only in superior prediction performance but also are uncorrelated with the bias or confounder variables. The code is available at http://blinded_for_review/. A central challenge in practically all machine learning applications is the consideration of confounding biases. Confounders are extraneous variables that distort the relationship between the input (independent) and output (dependent) variables and hence lead to erroneous . In a variety of applications ranging from disease prediction to face recognition, where machine learning models are built to predict labels from images, demographic variables (such as age, sex, race) of the study may confound the training process if the distribution of image labels is skewed with respect to them. In this situation, the predictor may learn the influence of the confounder and bias present in the data instead of actual discriminative cues. It is a cumbersome task to account for all biases when curating large-scale datasets . An alternative approach is to account for the bias in the model. Traditionally, confounding variables are often controlled by statistical methods in either design or analytical stages . In the design stage, one can utilize randomization or matching of the confounding variables across different study groups. In the analytical stage, confounding can be controlled by standardization or stratification . Another common solution is to learn the influence of the confounding variables on the input (independent) variables by regression analysis. Then, the residuals derived from the optimal regression model are regarded as the confounder-free input to train the predictor . The regression analysis works reasonably well under the assumption that the input variables represent deterministic features that are comparable across a population, e.g., morphometric measurements extracted from medical images or engineered features extracted from face images. The method fails, however, when this assumption does not hold such as for the pixel intensity values in images. Note, the raw intensities are only meaningful within a neighborhood but variant across images. Therefore, these regression approaches cannot be used in connection with deep learning methods that are di- † indicates equal contribution. Figure 1: Average face images across each shade category (first row), average saliency map of the trained baseline (second row), and BR-Net (third row) color-coded with the normalized saliency value for each pixel. BR-Net in more stable patterns across all 6 shade categories. The last column shows the tSNE projection of the learned features by each method. Our method in a better feature space invariant to the bias variable (shade) while the baseline shows a clear pattern affected by the bias. Average accuracy of per-shade gender classification over 5 runs of 5-fold crossvalidation is shown on each average map. The models are pre-trained on ImageNet and fine-tuned on GS-PPB. BR-Net is not only able to close the gap of accuracy for the darker shade but it also regularizes the model to improve per-category accuracy. rectly applied to images, such as convolutional neural networks (CNNs). Removing confounding factors for CNNs is an open question we aim to address here. We propose a feature learning scheme to produce features that are predictive of class labels while being unbiased to confounding variables. The idea is inspired by the domain-adversarial training approaches with controllable invariance within the context of generative adversarial networks (GANs) , but we argue that generic and widely used loss functions are not designed for controlling the invariance with respect to bias variables. Hence, we introduce an adversarial loss function that aims to quantify the statistical dependence between the learned features and bias variables with the correlation coefficient. This strategy improves over the commonly used cross-entropy or mean-squared error (MSE) loss that only aims to predict the exact value of the bias variables and thereby achieves stabler within the context of adversarial training. Since our proposed model injects resilience towards the bias during training to produce confounder-invariant features, we refer to our approach as Bias-Resilient Neural Network (BR-Net). We evaluate BR-Net on three datasets to examine different aspects of the method and compare it with a wide range of baselines. First, we test on a synthetic dataset to outline how the learned features by our method are unbiased to controlled confounding variables. Then, we test it on a medical imaging application, i.e., predicting the human immunodeficiency virus (HIV) diagnosis directly from T1-weighted Magnetic Resonance Images (MRIs). As widely explored in the HIV literature, HIV disease accentuates brain aging and if a predictor is learned not considering age as a confounder, the predictor may actually be learning the brain aging patterns rather than actual HIV markers. Lastly, we evaluate BR-Net for gender classification using the Gender Shades Pilot Parliaments Benchmark (GS-PPB) dataset . We use different backbones pre-trained on ImageNet and fine-tune them for predicting gender from face images. We show that prediction of the vanilla models is dependent on the race of the subject (alternatively we consider skin color quantified by the 'shade' variable) and show poor for darker faces, while BR-Net can successfully close the gap. Our comparison with methods based on multi-task prediction (i.e., predicting gender and shade as two tasks) and categorical GAN (i.e., predicting shade as a categorical variable in the adver-sarial component) shows that BR-Net is not only able to learn features impartial to the bias of race (verified by feature embedding and saliency visualization), it also in better performance in gender prediction (see Fig. 1). Fairness in Machine Learning: In recent years, developing fair machine learning models have been the center of many discussions (; ;) even in the news outlets and media . It has been argued that the bias essentially comes from human or society biases induced by the training datasets . Recent efforts in solving this problem have been focused on building fairer and more diverse datasets . However, this approach is not always practical for large-scale datasets or especially in medical applications, where data is relatively scarce and expensive to generate. In this work, we propose to use existing sets of data but to build models mindful of biases by learning impartial features that are only predictive of the actual output variables. Domain-Adversarial Training: proposed for the first time to use adversarial training for domain adaptation tasks by creating a component in the network that uses the learned features to predict which domain the data is coming from (a binary variable; source or target). Ever since, several other works built on top of the same idea explored different loss functions , domain discriminator settings, or cycle-consistency. The focus of all these works was to close the domain gap, which is often encoded as a binary variable. To learn generic bias-resilient models, we argue that we need to go beyond this and learn features that are invariant to either discrete or continuous confounders. There have been different attempts in the literature for learning representations that are invariant to specific factors in the data. For instance, took an information obfuscation approach to obfuscate membership in the protected group of data during training, and introduced a regularization-based method. Recently, (; ; ;) proposed to use domain-adversarial training strategies for controllable invariant feature learning with respect to existing variables in the data. Some concurrent works have also used adversarial techniques for mitigating the effect of bias. These methods used similar adversarial loss functions as in domain adaptation that aim at predicting exact values of the bias variables. For instance, used a binary cross-entropy for removing effect of'gender' and used linear (and kernelized) least squares predictors as the adversarial component. Our study shows that these strategies fail at creating resilience against biases that take continuous or ordinal values. Instead, we introduce a loss function based on correlation coefficient to naturally alleviate the bias effects on the learned features. Distribution Matching: Some previous work attempted to learn distributionally robust techniques to avoid learning confounded effects from data . This can be done by matching the distributions of the data across different domains. However, distribution matching techniques only model data of a population as a whole and fall short when it is crucial to remove the association between the learned features and a specific bias or confounding variable for each single input data point. Whereas, to close the gap with respect to the underlying bias in the data, our correlation-based analysis minimizes the bias-predictive power of the learned features for every individual data point, which by construction harmonizes the data distribution on the population level. Suppose we have an M -class classification problem, for which we have N pairs of training images and their corresponding target label(s):. Assume that the study is confounded or biased by a set of k variables, denoted by a vector b ∈ R k. To train a deep neural network for classifying each image to its label(s) while not being biased by the confounders in the study, we propose our end-to-end architecture as in Fig. 2 similar to domain-adversarial training approaches . Given the input image X, we first apply a Feature Extraction (FE) network, ing in a feature vector F. A Classifier (C) is built on top of this feature vector to predict the class label y for the input X, and it forces FE to learn discriminative futures for the classification task. Now, to guarantee that these features are not biased to b, we build another network (denoted by BP) with a new loss function for predicting the bias variables from F. We propose to backpropagate this loss to the feature extraction module in an adversarial way. As a , the feature extractor learns features that minimize the classification loss, while maximizing the bias predictor loss. Each network has its underlying trainable parameters, defined as θ f e for FE, θ c for C, and θ bp for BP. If the predicted probability that subject i belongs to class m is defined byŷ im = C(FE(X i ; θ f e); θ c ), the classification loss can be characterized by a cross-entropy: Similarly, withb i = BP(FE(X i ; θ f e); θ bp ), we can define the adversarial component of the loss function. Standard methods for designing this loss function suggest to use a cross-entropy for binary/categorical variables (e.g., in ) or an 2 MSE loss for continuous variables. However, we argue that in the context of bias control, the ultimate goal of adversarial training is to remove statistical association with respect to the bias variables, as opposed to maximizing the prediction error of them. In fact, the adversarial training based on MSE leads to the maximization of the 2 distance betweenb and b, which could be trivially achieved by uniformly shifting the magnitude ofb, thereby potentially ing in an ill-posed optimization and oscillation in the adversarial training. To address this issue, we define the following surrogate loss for predicting the bias confounders while quantifying the statistical dependence with respect to b: where corr 2 (·, ·) is the squared Pearson correlation between its inputs and b κ defines the vector of κ th bias variable across all inputs. Through adversarial training, we aim to remove statistical dependence by encouraging a zero correlation between b κ andb κ. Note, BP deems to maximize squared correlation and FE minimizes for it; Since corr 2 is bounded in the range, both minimization and maximization schemes are deemed feasible. Having these loss functions defined, the overall objective of the network is then defined as where hyperparameter λ controls the trade-off between the two objectives. This scheme is similar to GAN and domain-adversarial training , in which a min-max game is portrayed between two networks. In our case, FE extracts features that minimize the classification criterion, while fooling BP (i.e., making BP incapable of predicting the bias variables). Hence, the saddle point for this optimization objective is obtained when the parameters θ f e minimize the classification loss while maximizing the loss of the bias prediction module. Simultaneously, θ c and θ bp minimize their respective network losses. In general, a zero-correlation or a zero-covariance only quantifies linear independence between variables but cannot infer non-linear relationships. However, we now theoretically show that, under certain assumptions on the adversarial training of BP, a zero-covariance would guarantee the mean independence between bias variables and features, a much stronger type of statistical independence than the linear type. A random variable B is said to be mean independent of F if and only if for all ξ with non-zero probability, where E[·] defines the expected value. In other words, the expected value of B is neither linearly nor non-linearly dependent on F, but the variance of B might. The following theorem then relates the mean independence between features F and bias variables B to the zero-covariance between B and the prediction ofB produced by the adversarial component, BP. Property 1: B is mean independent ofB ⇒ Cov(B,B) = 0. Property 2: B, F are mean independent ⇒ B is mean independent ofB = φ(F) for any mapping function φ. Proof. The forward direction ⇒ follows directly through Property 1 and 2. We focus the proof on the reverse direction. Now, construct a mapping functionB = φ(2 holds if and only ifB is a constant, i.e., by definition, B is mean independent of F. Remark. In practice we normalize the covariance by standard deviations of variables for optimization stability. In the unlikely singular case that BP outputs a constant prediction, we add a small perturbation in computing the standard deviation. This theorem echoes the validity of our adversarial training strategy: FE encourages a zerocorrelation between b κ andb κ, which enforces b κ to be mean independent of F (one cannot infer the expected value of b κ from F). In turn, assuming BP has the capacity to approximate any arbitrary mapping function, the mean independence between features and bias would correspond to a zero-correlation between b κ andb κ, otherwise BP would adversarially optimize for a mapping function that increases the correlation. Similar to the training of GANs, in each iteration, we first back-propagate the L c loss to update θ f e and θ c. With θ f e fixed, we then minimize the L bp loss to update θ bp. Finally, with θ bp fixed, we maximize the L bp loss to update θ f e. The last step can be considered as the bias effect removal component. Furthermore, in the present study, L bp depends on the correlation operation, which is a population-based operation, as opposed to individual-level error metrics such as cross-entropy or MSE losses. Therefore, we calculate the correlations over each training batch as a batch-level operation. Depending on the application, we can use different architectures for each of the three subnetworks. We use a 3D CNN for FE to extract features from 3D medical images and use VGG16 and ResNet50 backbones for GS-PPB. For C and BP, we use a two-layer fully connected network. In this section, we evaluate our method on three different scenarios. First, we run a synthetic experiment to verify the validity of our assumptions. Then, we apply BR-Net to predict diagnosis of HIV from brain MRIs confounded by the subjects' age. Finally, we test the model for predicting gender from face images and show how controlling for variables related to race (e.g., face color shade) can robustly enhance prediction performance. We compare BR-Net with several baselines, and evaluate how the features learned by our method are invariant to the bias or confounding variables. Baseline Methods. In line with the implementation of our approach, the baseline for all three experiments is a vanilla CNN, whose architecture is exactly the same as BR-net except that there is no bias prediction sub-network and hence the adversarial loss. We emphasize that BR-Net aims to remove the association between prediction and bias by encouraging vanished correlation, which is different from simply maximizing the prediction loss (w.r.t bias) as usually performed in many GAN settings. Therefore, the second comparison method is a BR-Net with the adversarial loss being the MSE, denoted by'BR-Net (w/ MSE).' For the Gender Shades PPB experiment, we further add two other baseline methods, one predicting both'gender' and'shade' in a multi-task setting , denoted by'Multi-Task'; and one replacing correlation loss function L bp with a cross-entropy loss as the'shade' variable has a ordinal but categorical value. The adversarial training then relies on maximizing the entropy of BP predictions as motivated in Categoral GAN models ('CatGAN') . These baselines show how the correlation loss plays an important role in delineating the bias and confounding effects. We generate a synthetic dataset comprised of two groups of data, each containing 512 images of resolution 32 × 32 pixels. Each image is generated by 4 Gaussians (see Fig. 3a), the magnitude of which is controlled by σ A and σ B. For each image from Group 1, we sample σ A and σ B from a uniform distribution U while we generate images of Group 2 with stronger intensities by sampling from U. Gaussian noise is added to the images with standard deviation 0.01. Now we assume the difference in σ A between the two groups is associated with the true discriminative cues that should be learned by a classifier, whereas σ B is a given confounder. In other words, an unbiased model should predict the group label purely based on the two diagonal Gaussians and not dependent on the two off-diagonal ones. To show that the BR-Net can in such models by controlling for σ B, we train it on the whole dataset of 1,024 images given their respective binary labels and confounder values σ B. For simplicity, we construct the FE Network with 3 stacks of 2 × 2 convolution/ReLU/max-pooling layers to produce 32 features. Both the BP and C networks have one hidden layer of dimension 16 with tanh as the non-linear activation function. After training, BR-Net achieves 89% training accuracy and BR-Net w/ MSE achieves 90%. Note that the theoretically maximum training accuracy is 90% due to the overlapping sampling range of σ A between the two groups. The baseline model, however, achieves 95% accuracy, indicating that the model additionally relies on the confounding effects σ B for predicting the group label, an undesired behavior. To further investigate the association between the learned features and σ B, we measure their squared distance correlation (dcor 2) (Székely et al., 2007) for the training samples in Group 1 (when there is no association between σ B and prediction, dcor=0 for either group). Distance correlation is a widely-used measure of dependence between two paired vectors of arbitrary dimensions. Fig. 3b shows that our method successfully removes the statistical association w.r.t σ B as the distance correlation drops dramatically with training iterations. On the other hand, the baseline model without the BP component learns features that constantly yield high correlation. Note that the adversarial loss based on MSE yields unstable dcor 2 measures potentially due to the ill-posed optimization of maximizing 2 distance. Finally, the above are further supported by the 2D tSNE projection of the learned features as shown in Fig. 3c. The feature space learned by the baseline model forms a clear correlation with σ B, whereas our method in a space with no apparent bias. This confirms that the proposed adversarial technique successfully removes the bias from the confounding variable. Our second task aims at predicting the diagnosis of HIV patients vs.control subjects (CTRL) based on brain MRIs. The study cohort includes 223 CTRLs and 122 HIV patients who are seropositive for the HIV-infection with CD4 count > 100 cells µL (average: 303.0). Since the HIV subjects are significantly older in age than the CTRLs (CTRL: 45 ± 17, HIV: 51 ± 8.3, p < .001) in this study, age becomes a potential confounder; prediction of diagnosis labels may be dependent on subjects' age instead of true HIV markers. The T1-weighted MRIs are all skull stripped, affinely registered to a common template, and resized into a 64 × 64 × 64 volume. Classification accuracy is measured with 5-fold cross validation. For each run, the training folds are augmented by random shifting (within one-voxel distance), rotation (within one degree) in all 3 directions, and left-right flipping based on the assumption that HIV infection affects the brain bilaterally. The data augmentation in a balanced training set of 1024 CTRLs and 1024 HIVs. As the flipping removes left-right orientation, the ConvNet is built on half of the 3D volume containing one hemisphere. The feature extractor FE has 4 stacks of 2×2×2 3D convolution/ReLu/batch-normalization/max-pooling layers yielding 4096 intermediate features. Both BP and C have one hidden layer of dimension 128 with tanh as the activation function. For this experiment, as suggested in the previous work , confounding effects can only be reliably estimated among healthy subjects. So, in practice we only perform the adversarial loss back-propagation step for the CTRL group. Each point shows a subject in the CTRL cohort color-coded by their age. Table 1 shows the diagnosis prediction accuracy of BR-Net in comparison with 3D CNN, BRNet (w/ MSE), and Resid+SVM (note, to compare with the traditional residualization methods, we extract 298 brain regional measurements, residualize the confounders using a general linear model, and classify with a support vector machine). Our method (BR-Net) in the most accurate prediction in terms of balanced accuracy (bAcc), area under curve (AUC), and F 1 -score from the cross-validation. These show that our method is able to learn discriminative features while controlling for confounders. In addition, we record the balanced accuracy, true positive, and true negative rate for each training iteration. As shown in Fig. 5, the baseline tends to predict most subjects as CTRLs (high true negative rate). This is potentially caused by the CTRL group having a wider age distribution, so an age-dependent predictor would bias the prediction towards CTRL. On the other hand, when controlling age as a confounder, BR-Net reliably in balanced true positive and true negative rates. Similar to the previous experiment, we train different methods on the entire dataset and plot the squared distance correlation between the learned features and confounders for the CTRL cohort (Fig. 4). The figure shows that for BR-Net the distance correlation between the features and the confounding variable (age) decreases with the adversarial training. Whereas, the baseline model 3D CNN consistently produces features that are highly correlated with the confounder, and BR-Net w/ MSE produces inconsistent and unreliable associations with respect to the confounder. The t-SNE projection of the learned feature spaces are visualized in Fig. 6. The feature space learned by the baseline model forms a clear association with age, as older subjects are concentrated on the top left region of the space. This again suggests predictions from the baseline may be dependent on age rather than true HIV markers. Whereas, our method in a space with no apparent bias to age. The last experiment is on gender prediction from face images in the Gender Shades Pilot Parliaments Benchmark (GS-PPB) dataset . This dataset contains 1,253 facial images of 561 female and 692 male subjects. The face shade is quantified by the Fitzpatricksix-point labeling system and is categorised from type 1 (lighter) to type 6 (darker). This quantization was used by dermatologistsuse for skin classification and determining risk for skin cancer . To train our models on this dataset, we use backbones VGG16 and ResNet50 pre-trained on ImageNet . We fine-tune each model on GS-PPB dataset to predict the gender of subjects based on their face images using fair 5-fold cross-validation. The ImageNet dataset for pre-training the models has fewer cases of humans with darker faces , and hence the ing models have an underlying bias to the shade. BR-Net counts the variable'shade' as an ordinal and categorical bias variable. As discussed earlier, besides the vanilla VGG16 and ResNet50 models, we compare the with a multi-task baseline , which predicts both'gender' and'shade' simultaneously, and a model that uses the entropy loss as the adversarial loss for the cross-entropy-based categorical prediction (proposed by CatGAN ). Table 2 shows the across five runs of 5-fold cross-validation. Fig. 7 plots the accuracy for each individual'shade' category. As can be seen from the table and Table 2: Average over five runs of 5-fold cross-validation on the GS-PPB dataset. Best in each column are typeset in bold. the figure, BR-Net outperforms other methods on average while producing similar accuracy across all'shade' categories. Prediction made by other methods, however, is more dependent on the bias variable by showing inconsistent recognition capabilities for different'shade' categories and failing significantly on darker faces. This bias is confirmed by the tSNE projection of the feature spaces learned by different methods (see Fig. 8). The features learned by the vanilla baseline or even the multi-task model show a clear dependency on the'shade' while BR-Net in a roughly uniform distribution of subjects. To gain more insight into the , we visualize the saliency maps derived for the baseline and BR-Net. For this purpose, we use a similar technique as in to extract the pixels in the original image space highlighting the areas that are discriminative for the gender labels. Generating such saliency maps for all inputs, we visualize the average map for each individual'shade' category (Fig. 1). The value on each pixel corresponds to the attention from the network to that pixel within the classification process. Compared to the baseline, BR-Net focuses more on specific face regions and in more stable patterns across all'shade' categories. We proposed a method based on adversarial training strategies by encouraging vanished correlation to learn features for the prediction task while being unbiased to the confounding variables in the study. We evaluated our bias-resilient neural network (BR-Net) on a synthetic, a medical diagnosis, and a gender prediction dataset. In all experiments, BR-Net ed in a feature embedding space that was agnostic to the bias in the data while all other methods failed to do so. Based on our experiments we can conclude that, besides the attempt to improve datasets and curate unbiased ones , it is crucial to build models that properly account for the bias in data during training. Our bias-resilient model and some other recent works set on foot toward this direction. This is crucial as machine learning models are acceding to everyday lives, or are being developed for crucial medical applications. Failure to account for the underlying bias or confounding effects can lead to spurious associations and erroneous decisions. As a direction for the future work, other strategies such as deep canonical correlation analysis can be explored to form the adversarial component.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the article for finding a single-sentence TLDR

Question: Existing neural question answering (QA) models are required to reason over and draw complicated inferences from a long context for most large-scale QA datasets. However, if we view QA as a combined retrieval and reasoning task, we can assume the existence of a minimal context which is necessary and sufficient to answer a given question. Recent work has shown that a sentence selector module that selects a shorter context and feeds it to the downstream QA model achieves performance comparable to a QA model trained on full context, while also being more interpretable. Recent work has also shown that most state-of-the-art QA models break when adversarially generated sentences are appended to the context. While humans are immune to such distractor sentences, QA models get easily misled into selecting answers from these sentences. We hypothesize that the sentence selector module can filter out extraneous context, thereby allowing the downstream QA model to focus and reason over the parts of the context that are relevant to the question. In this paper, we show that the sentence selector itself is susceptible to adversarial inputs. However, we demonstrate that a pipeline consisting of a sentence selector module followed by the QA model can be made more robust to adversarial attacks in comparison to a QA model trained on full context. Thus, we provide evidence towards a modular approach for question answering that is more robust and interpretable.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

A modular approach consisting of a sentence selector module followed by the QA model can be made more robust to adversarial attacks in comparison to a QA model trained on full context.

Represent the short summary for finding the corresponding scientific article (long):

We revisit self-training as a semi-supervised learning method for neural sequence generation problem, and show that self-training can be quite successful with injected noise.


Question: Multi-relational graph embedding which aims at achieving effective representations with reduced low-dimensional parameters, has been widely used in knowledge base completion. Although knowledge base data usually contains tree-like or cyclic structure, none of existing approaches can embed these data into a compatible space that in line with the structure. To overcome this problem, a novel framework, called Riemannian TransE, is proposed in this paper to embed the entities in a Riemannian manifold. Riemannian TransE models each relation as a move to a point and defines specific novel distance dissimilarity for each relation, so that all the relations are naturally embedded in correspondence to the structure of data. Experiments on several knowledge base completion tasks have shown that, based on an appropriate choice of manifold, Riemannian TransE achieves good performance even with a significantly reduced parameters. 1.1 Multi-relational graphs, such as social networks and knowledge bases, have a variety of applications, and embedding methods for these graphs are particularly important for these applications. For instance, multi-relational graph embedding has been applied to social network analysis and knowledge base completion BID2. A multi-relational graph consists of entities V, a set R of relation types, and a collection of real data triples, where each triple (h, r, t) ∈ V × R × V represents some relation r ∈ R between a head entity h ∈ V and a tail entity t ∈ V. Embedding a multi-relational graph refers to a map from the entity and the relation set to some space. Mathematical operations in this space enable many tasks, including clustering of entities and completion, prediction, or denoising of triples. Indeed, completion tasks for knowledge bases attract considerable attention, because knowledge bases are known to be far from complete, as discussed in BID13. Multi-relational graph embedding can help its completion and improve the performance of applications that use the graph. This is the reason why much work focuses on multi-relational graph embedding. FIG0 shows an example of a multi-relational graph and a completion task. In multi-relational graph embedding, reducing the number of parameters is an important problem in the era of big data. Many parameters are needed with tensor-factorization-based methods, such as Bayesian clustered tensor factorization (BCTF) , RESCAL , and a neural tensor network (NTN) , where each relation has a dense matrix or tensors (O D 2 or more parameters, where D is dimensionality of the space). Thus, TransE BID2 was proposed to reduce the number of parameters, to overcome this problem. In TransE, each entity is mapped to a point in Euclidean space and each relation is no more than a vector addition (O (D) parameters), rather than a matrix operation. The successors to TransE, TransH and TransD BID11, also use only a small number of parameters. Some methods succeeded in reducing parameters using diagonal matrices instead of dense matrices: e.g. DISTMULT , ComplEx , HolE (through the Fourier transform) , and ANALOGY BID15. In these methods, all relations share one space for embedding, but each relation uses its own dissimilarity criterion. The success of these methods implies that one common space underlies whole data, and each relation can be regarded as a dissimilarity criterion in the space. Whereas these methods use distances or inner products in Euclidean space as dissimilarity criteria, recent work has shown that using non-Euclidean space can further reduce the number of parameters. One typical example of this is Poincaré Embedding for hierarchical data, where a hyperbolic space is used as a space for embedding. Here, the tree structure of hierarchical data has good compatibility with the exponential growth of hyperbolic space. Recall the circumference with radius R is given by 2π sinh R(≈ 2π exp R) in a hyperbolic plane. As a , Poincaré embedding achieved good graph completion accuracy, even in low dimensionality such as 5 or 10. On the other hand, spheres (circumference: 2π sin R) are compatible with cyclic structures. Since Poincaré embedding, several methods have been proposed for single-relational graph embedding in non-Euclidean space (e.g. BID8, ) and shown good . The success of these methods suggests that the appropriate choice of a manifold (i.e., space) can retain low dimensionality, although these methods are limited to single-relational graph embedding. According to the success of the TransE and its derivation and Poincaré embedding, it is reasonable in multi-relational graph embedding to assume the existence of a single structure compatible with a non-Euclidean manifold. For example, we can consider a single tree-like structure, which contains multiple hierarchical structures, where root selection gives multiple hierarchical structures from a single tree, which is compatible with hyperbolic spaces (See Figure 2). Therefore, embedding in a single shared non-Euclidean manifold with multiple dissimilarity criteria used in TransE is promising. Taking Poincaré embedding's success with low dimensionality into consideration, this method should work well (e.g., in graph completion tasks) with small number of parameters. This is the main idea of this paper. There are five entities and two kinds of relation (hypernym and synonym). Graph completion refers to answering questions such as "is mammal a hypernym of cannis?"Figure 2: Multiple hierarchical relations in a single tree. As this example shows, it is possible that multiple relations are given by multiple dissimilarity criteria in a single structure. We propose a novel method, called Riemannian TransE, for multi-relation graph embedding using a non-Euclidean manifold. In Riemannian TransE, the relations share one non-Euclidean space and the entities are mapped to the space, whereas each relation has its own dissimilarity criterion based on the distance in the space. Specifically, the dissimilarity criteria in Riemannian TransE are similar to those in TransE BID2 ) based on vector addition, which is known to be effective. Unfortunately, we cannot straightforwardly use TransE's dissimilarity criteria. This is due to non-existence of a parallel vector field (See Figure 4), which is implicitly but essentially used in "vector addition." However, the parallel condition is not essential in TransE's idea. For example, hierarchical bottom to top relations should be regarded as attraction to the top in the hierarchy, which is not parallel but has an attractive point. Moreover, parallel vector fields can be regarded as a vector field attracted to a point at infinity. Therefore, we replace parallel vector fields in TransE by vector fields with an attractive point that are well-defined in Riemannian manifolds, and as a , we obtain Riemannian TransE. Advantages of non-Euclidean spaces enable our Riemannian TransE to achieve good performance (e.g. in graph completion) with low-dimensional parameters. Riemannian TransE further exploits the advantages of TransE: that is, the method needs only O (D) parameters for each relation. Numerical experiments on graph completion tasks show that with an appropriate choice of manifold, our method can improve the performance of multi-relational graph embedding with few parameters. Let V and R denote the entities and relations in a multi-relational graph, and let T ⊂ V × R × V denote the triples in the graph. Multi-relational graph embedding refers to a pair of maps from V and R into M e and M r, respectively. Particularly, learning multi-relational graph embedding refers to obtaining an appropriate pair of maps v → p v (v ∈ V, p v ∈ M e) and r → w r (r ∈ R, w r ∈ M r) from the triples T. In this paper, we call p v the planet of entity v, w r the launcher of relation r, and M e and M r the planet manifold and launcher manifold, respectively. The quality of embedding is measured through a score function f: (M e × M e) × M r → R, which is designed by each method. Embedding is learned such that the value score function f (p h, p t ; w r) will be low when p h, p t; w r ∈ T and high when p h, p t; w r / ∈ T. For specific loss functions designed from the score function, see Subsection 2.3. We interpret the score function of multi-relational graph embedding as dissimilarity in a manifold, which we call a satellite manifold M s. We rewrite the score function f in multi-relational graph embedding using two maps H, T: M e × M r → M s and the dissimilarity measure function D: M s × M s → R as follows: DISPLAYFORM0 We call H and T the head and tail launch map, respectively, and call s H v;r and s T v;r the head and tail satellite of entity v (or of planet p v) with respect to relation r. The idea of this formulation is embedding in one shared space with multiple dissimilarity criteria. Specifically, each entity has only one planet and their satellite pairs give multiple dissimilarity criteria, each of which corresponds to a relation. In other words, all of the relations shares one space and the planets in it, and the differences among the relations are reduced to the difference of their launcher maps and the satellites given by them. We regard the planets as the embeddings of the entities, whereas dissimilarity between entities with respect to a relation is evaluated through their satellites which correspond to the relation. A simple example of this is TransE BID2, where all of the planets, satellites, and launchers share the same Euclidean space, i.e. M e = M s = M r = R D, the launch maps are given by vector addition as H (p; w) = p + w and T (p; w) = p, and the distance in a norm spacei.e. the norm of the difference-is used as a dissimilarity criterion i.e. D s DISPLAYFORM1 (the L1 or L2 norm is often used in practice). See Figure 5 (left). suggested, one can associate the idea of representing relations as vector additions with the fact that we can find a relation through a substraction operator in Word2Vec. That is, we can find relations such as p France − p Paris ≈ p Italy − p Rome in Word2Vec. As explained above, TransE is based on the distance between satellites, and each satellite is given by simple vector addition. Regardless of this simplicity, the performance of TransE has been exemplified in review papers (. Indeed, the addition operation in a linear space is essential in the launcher map, and hence TransE can easily be extended to a Lie group, which is a manifold equipped with an addition operator, as suggested in BID5 . Some methods, such as TransH , TransR BID14, and TransD BID11, also use a norm in linear space as a dissimilarity measure, integrating a linear map into a latent space. Another simple example is RESCAL , which uses the negative inner product as a dissimilarity measure. In RESCAL, the launcher of relation r is a matrix W ∈ M r = R D×D, the launch maps are given by a linear map, i.e. H (p; (W, w)) = W p and T (p; (W, w)) = p, and the dissimilarity measure is the negative inner product D s DISPLAYFORM2 Other methods are also based on the (negative) inner product dissimilarity: e.g., DISTMULT , ComplEx , HolE (through the Fourier transform) , and ANALOGY BID15. Table 1 shows score functions of these methods. Whereas some methods are based on a neural network (e.g., the neural tensor network and ConvE BID3 ), their score function consists of linear operations and element-wise nonlinear functions. Graph embedding using non-Euclidean space has attracted considerable attention, recently. Specifically, embedding methods using hyperbolic space have achieved outstanding (Nickel & Kiela, Table 1 : Score Functions. The launcher w r of r determines the dissimilarity criterion of r through satellites. In this table, the dimensionality is set so that the (real) dimensionality of the planets is D. † denotes conjugate transpose. F denotes the discrete Fourier Transform. The interpretation here of HolE is given by BID15 and BID10 BID8 ) (. With these methods, each node in the graph is mapped to a point in hyperbolic space and the dissimilarity is measured by a distance function in the space. Although these methods exploit the advantages of non-Euclidean space, specifically those of a negative curvature space, they focus on single-rather than multi-relational graph embedding. DISPLAYFORM0 By contrast, TransE has been extended to an embedding method in a Lie group-that is, a manifold with the structure of a group BID5 . As such, the regularization problem in TransE is avoided by using torus, which can be regarded as a Lie group. Although this extension to TransE deals with multi-relational embedding, it cannot be applied to all manifolds. This is because not all manifolds have the structure of a Lie group. Indeed, we cannot regard a hyperbolic space (if D = 1) or a sphere (if D = 1, 3) as a Lie group. We can simply design a loss function on the basis of the negative log likelihood of a Bernoulli model as follows: DISPLAYFORM0 DISPLAYFORM1 where Q is the set of the triples with its corrupted head and tail. That is, DISPLAYFORM2 where δ ∈ R ≥0 is the margin hyperparameter, and [·] + denotes the negative value clipping-i.e. for all x ∈ R, [x] +:= max(x, 0). We use this loss function throughout this paper. In this section, we formulate Riemannian TransE exploiting the advantages of TransE in nonEuclidean manifolds. Firstly, we give a brief introduction of Riemannian geometry. Secondly, we explain the difficulty in application of TransE in non-Euclidean manifolds. Lastly, we formulate Riemannian TransE. Let (M, g) be a Riemannian manifold with metric g. We denote the tangent and cotangent space of M on p by T p M and T * p M, respectively, and we denote the collection of all smooth vector DISPLAYFORM0 denote the LeviCivita connection, the unique metric-preserving torsion-free affine connection. A smooth curve γ: (−,) → M is a geodesic when ∇γγ = 0 on curve γ, whereγ is the differential of curve γ. Geodesics are generalizations of straight lines, in the sense that they are constant speed curves that are locally distance-minimizing. We define the exponential map Exp p, which moves point p ∈ M towards a vector by the magnitude of the vector. In this sense, the exponential map is regarded as an extension of vector addition in a Riemannian manifold. FIG2 shows an intuitive example of an exponential map on a sphere. Let DISPLAYFORM1 We define the logarithmic map Log p: M → T p M as the inverse of the exponential map. Note that the exponential map is not always bijective, and we have to limit the domain of the exponential and logarithmic map appropriately, while some manifolds, such as Euclidean and hyperbolic space, do not suffer from this problem. In TransE, a single vector w r determines the head and tail launch maps H, T as a transform: DISPLAYFORM0 In fact, these launch maps are given by vector addition. Note that this constitution of the launcher maps implicitly but essentially uses the fact that a vector is identified with a parallel vector field in Euclidean space. Specifically, a vector w determines a parallel vector field, denoted by W r here, which gives a tangent vector [W r] p ∈ T p R D on every point p ∈ R D, and each tangent vector determines the exponential map Exp p ([W r] p ) at p, which is used as a launch map in TransE. However, because there is no parallel vector field in non-zero curvature spaces, we cannot apply TransE straightforwardly in non-zero curvature spaces. Thus, extention of TransE in non-Euclidean space non-trivial. This is the difficulty in Riemannian Manifolds. As we have explained in Introduction, our idea is replacing parallel vector fields in TransE by vector fields attracted to a point. Specifically, we obtain the Riemannian TransE as an extension of TransE, replacing the launchers w r ∈ R D in TransE by pairs w r = (r, p r) ∈ R × M of a scalar value and point, indicating the length and destination of the satellites' move, respectively. We call p r the attraction point of relation r. In other words, we replace parallel vector field W r = w r in TransE by r DISPLAYFORM0 Note that, we use a fixed manifold M e = M for entity embedding and use direct product manifold M r = R × M for relation embedding. However, the extension still has arbitrariness. For instance, we could launch the tail satellite instead of the head satellite in TransE; in other words, the following launching map also gives us a score function equivalent to that of the original TransE: H (p; w) = p and T (p; w) = p − w (Figure 5 center). On the other hand, the score function depends on whether we move the head or tail satellites In these examples, the number |V| of entities is three and the number |R| of relations is two (red and orange), with triples (1, orange, 2) and (1, red, 3). Hence, these models learn that the orange head satellite of Entity 1 is close to the orange tail satellite of Entity 2 and the red head satellite of Entity 1 is close to the red tail satellite of Entity 3. In addition, the distance of the other pair of satellites should be long in the representation learned by each method. The figure on the left shows the original formulation of TransE, where the satellites are given by vector addition. In other words, the satellites are given by a move towards a point at infinity from the planet. The center figure shows an alternative formulation of TransE, which is equivalent to the original TransE. Here, the tail satellites are launched and the head satellites are fixed in the red relation. In Riemannian TransE in the figure on the right, the vector additions are replaced by a move towards a (finite) point. Figure 6: Relation of the sign for. If is positive (e.g. the orange relation), the relation runs from low (e.g. Entity 2 and 3) to high hierarchy (e.g. Entity 1), and vice versa (e.g. the red relation).in our case, where the attraction points are not at infinity. With hierarchical data, an entity at a higher hierarchy has many related entities in a lower hierarchy. Therefore, it is best to always launch the satellites of "children," the entities in a lower hierarchy, toward their parent. Hence, we move the head satellites when r > 0 and fix the tail satellites, and vice versa when r < 0; specifically, we move the head satellites by length λ = [r] + and move the tail satellites by length λ = [− r] +. Thus, bottom-to-top relation cases correspond to r > 0 (Figure 6, left), and top-to-bottom relation cases correspond to r < 0 (Figure 6, right). Another problem pertains to launching the satellites near the attraction point. If λ > ∆ (p r, p v), the naive rule causes overrun. In this case, we simply clip the move and set the satellite in the place of p r.We turn now to the score function of Riemannian TransE. The score function f: (M × M) × (R, M) → R in Riemannian TransE is given as follows: DISPLAYFORM1 where transform m λ,p: M → M denotes a move, defined as follows: DISPLAYFORM2 Here, note that m,p (q) is on the geodesic that passes through p and q. Figure 5 (right) shows the Riemannian TransE model. If M = R D and the attraction points are at infinity, the score function is equivalent to that of TransE (without the sphere constraint). Although the exponential map and logarithmic map in closed form are required to implement Riemannian TransE, we can obtain them when the manifold M is a sphere S D (positive curvature), Euclidean space R D (zero curvature), and hyperbolic space H D (negative curvature), or a direct product of them. These are practically sufficient. Also note that the computation costs of these maps are O(D), which is small enough. In typical cases, the number of entities is very large. Therefore, stochastic gradient methods are effective for optimization. Although we can directly apply stochastic gradient methods of Euclidean space or the natural gradient method , Riemannian gradient methods (e.g. ) work better for non-Euclidean embedding BID6. In this paper, we use stocastic Riemannian sub gradient methods with norm clipping (See Appendix). Note that in spheres or hyperbolic spaces, the computation costs of the gradient is O(D), which is as small as TransE. Evaluation Tasks We evaluated the performance of our method for a triple classification task on real knowledge base datasets. The triple classification task involved predict-ing whether a triple in the test data is correct. We label a triple positive when f (p h, p t ; ( r, p r)) > θ r, and vice versa. Here, θ r ∈ R ≥0 denotes the threshold for each relation r, which is determined by the accuracy of the validation set. We evaluated the accuracy of classification with the FB13 and WN11 datasets . Although we do not report the of link prediction tasks BID2 here because there are many evaluation criteria for the task, which makes it difficult to interpret the , we report the in Appendix. BID15. We used implementations of these methods on the basis of OpenKE http://openke.thunlp.org/static/index.html, and we used the evaluation scripts there. Note that we compensated for some missing constraints (for example, in TransR and TransD) and regularizers (for example, in DISTMULT and Analogy) in OpenKE. We also found that omitting the constraint of the entity planets onto the sphere in TransE gave much better in our setting, so we also provide these unconstrained (UnconstraintTransE). We determined the hyperparameters by following each paper. For details, see the Appendix. Results TAB1 shows the for the triple classification task in each dimensionality. In WN11, the sphere-based Riemannian TransEs achieved good accuracy. The accuracy did not degrade dramatically even with low dimensionality. On the other hand, in FB13, the hyperbolic-space-based Riemannian TransEs was more accurate than other methods. Moreover for each dimensionality, these with the proposed Riemannian TransE were at least comparable to those of the baselines. The accuracy of Euclidean-space-based methods (e.g. the original TransE, and Euclidean TransE) are between that of the sphere-based Riemannian TransEs and that of the hyperbolic-spacebased Riemannian TransEs in most cases. Note that these are compatible with the curvature of each space (i.e. Sphere: positive, Euclidean space: 0, a hyperbolic space: negative). Note that Euclidean methods are sometimes better than non-Euclidean methods. In Appendix, we also report the triple classification task in FB15k, where Euclidean TransE as well as baseline methods outperformed Riemannian TransE did not always outperform the baseline methods. In summary, positive curvature spaces were good in WN11 and negative curvature spaces were good in FB13, and zero curvature spaces were good in FB15k. These show that Riemannian TransE can attain good accuracy with small dimensionality provided that an appropriate manifold is selected. What determines the appropriate manifold? Spheres are compatible with cyclic structure and hyperbolic spaces are compatible with tree-like structure. One possible explanation is that WN11 has cyclic structure and FB13 has tree-like structure and the structure of FB15k is between them. However, further discussion remains future work. We proposed Riemannian TransE, a novel framework for multi-relational graph embedding, by extending TransE to a Riemannian TransE. Numerical experiments showed that Riemannian TransE outperforms baseline methods in low dimensionality, although its performance depends significantly on the choice of manifold. Hence, future research shall clarify which manifolds work well with particular kinds of data, and develop a methodology for choosing the appropriate manifold. This is important work not only for graph completion tasks but also for furthering our understanding of the global characteristics of a graph. In other words, observing which manifold is effective can help us to understand the global "behavior" of a graph. Other important work involves using "subspaces" in non-Euclidean space. Although the notion of a subspace in a non-Euclidean manifold is nontrivial, it may be that our method offers advantages over TransH and TransD, which exploit linear subspaces. In this paper, we use the following simple (projected) stochastic (Riemannian) (sub-) gradient methods DISPLAYFORM0 where DISPLAYFORM1 |R| denotes the parameter in the τ -th step, η ∈ R ≥0 is the DISPLAYFORM2 |R| is a stochastic gradient that satisfies DISPLAYFORM3 Recall that denotes index raising. Specifically, we use the following stochastic loss function based on the mini-batch method: DISPLAYFORM4 where the stochastic quintet set Q (τ) ⊂ Q is a set of uniform-distributed random variables on Q. DISPLAYFORM5. We obtain a stochastic gradient as follows: DISPLAYFORM6 where θ is a local coordinate representation of θ. We obtain∇ (τ) easily using an automatic differentiation framework. Algorithm 1 shows the learning algorithm for Riemannian TransE. In the experiments, we applied norm clipping such that the norm of a stochastic gradient is smaller than 1. end for return θ (τ) We give additional explanations of the reason why we cannot define a parallel vector field on a non-Euclidean manifold. Specifically we describe the relationship between parallel vector fields and parallel transform. We can define a parallel transform along a geodesic. This parallel transform maps a tangent vector in a tangent space to one in another. At one glance, it seems that we can define a parallel vector field using the parallel transform. However, a parallel transform is not determined only by the origin and destination but depends on the path i.e. the geodesic. Figure 7 shows an example on a sphere, where two ways to map a vector from a tangent space to another are shown and these two give different maps. As this figure shows we cannot obtain a well-defined vector on more than two points. Figure 7: Parallel transforms in a sphere S 2. This figure shows two ways to transform vector v ∈ T p S 2 to T r S 2. We denote the parallel transform from along segment pq by Π q p: T p S 2 → T q S 2. The red vector on T r S 2 denotes the vector obtained by the direct transform along segment pr. The blue vector T r S 2 denotes the vector obtained by the transform via q. As this figure shows we cannot obtain a well-defined vector on more than two points. We introduces some Riemannian manifolds useful in applications, and the formula of the exponential map and logarithmic map in these manifolds. The closed form of exponential map and logarithmic map enables implementation of Riemannian TransE in these manifolds. In the following, we omit symbols ∂ ∂x and d x of the basis in a tangent and cotangent space, respectively, for notation simplicity. Moreover, we give the composition of the exponential map and index raising and that of the index lowering and logarithmic map instead of the exponential map and logarithmic map themselves. This is because we use a cotangent vector rather than a tangent vector in a practical implementation and map from/to cotangent space is more useful (Recall that ∂ ∂θ L is not the coordinate of a tangent but the coordinate of a cotangent vector). In a D-dimensional Euclidean Space, the exponential map (with the index raising) Exp p •: DISPLAYFORM0 Apparently, the logarithmic map (with the index lowering) DISPLAYFORM1 C.2 SPHERE A D-dimensional (unit) sphere is given by point set S D:= p ∈ R (D+1) p p = 1, and the DISPLAYFORM2 between two points p ∈ S D and q ∈ S D is given as follows: DISPLAYFORM3 where arccos: [−1, 1] → [0, π] denote arc-cosine function. The exponential map (with the index raising) Exp p •: DISPLAYFORM4 where sinc denotes the cardinal sine function defined as follows: DISPLAYFORM5 The logarithmic map (with the index lowering) DISPLAYFORM6 Note that in optimization, we need the projection of the differentialδ = ∂ ∂θ L (θ)| θ=p of the loss function L to cotangent vector δ given by: DISPLAYFORM7 In this subsection, we introduces models of a hyperbolic space, which are mathematically equivalent to each other, but have practically different aspects. There are many models of a hyperbolic space. We introduce two of them: the hyperboloid model and Poincaré disk model. Some formulae here are also given and used in. Let G M denote diagonal matrix DISPLAYFORM0 In the hyperboloid model, a (canonical) hyperbolic space is given by point set DISPLAYFORM1 p, δ = 0, and the metric g p: DISPLAYFORM2 The distance ∆ (p, q) between two points p ∈ H D and q ∈ H D is given as follows: DISPLAYFORM3 where, arcosh: [1, ∞) → [0, ∞) denotes the area hyperbolic cosine function, i.e. the inverse fucntion of the hyperbolic cosine function. The exponential map (with the index raising) Exp p •: DISPLAYFORM4 where sinhc denotes the hyperbolic sine cardinal function defined as follows: DISPLAYFORM5 The logarithmic map (with the index lowering) DISPLAYFORM6 D.1.1 LINK PREDICTION TASK In the link prediction task, we predict the head or the tail entity given the relation type and the other entity. We evaluate the ranking of each correct test triple (h, r, t) in the corrupted triples. We corrupt each triple as follows. In our setting, either its head or tail is replaced by one of the possible head or entity, respectively. In addition, we applied "filtered" setting proposed by BID2, where the correct triples, that is, the triples T in the original multi-relational graph are excluded. Thus, the corrupted triples are given by (h, r, t) h ∈ V h ∧ (h, r, t) / ∈ T (head corruption) or {(h, r, t) | t ∈ V t ∧ (h, r, t) / ∈ T } (tail corruption). where V h r and V t r denote the possible heads and tails in relation r, given as follows: DISPLAYFORM7 As evaluation metrics, we use the following:Mean rank (MR) the mean rank of the correct test triples. The value of this metric is always equal to or greater than 1, and the lower, the better. Hits @ n (@n) the propotion of correct triples ranked in the top n predictions (n = 1, 3, 10). The value ranges from 0 to 1, and the higher, the better. Mean reciprocal rank (MRR) the mean of the reciprocal rank of the correct test triples. The value ranges from 0 to 1, and the higher, the better. In triple classification tasks, we predict whether a triple in the test data is correct or not. The classification is simply based on the score function i.e. we label a triple positive when f (p h, p t ; ( r, p r)) > θ r, and the other way around. Here, θ r ∈ R ≥0 denotes the threshold for each relation r, which is determined by the accuracy in the validation set. In link prediction tasks, we used WN18 and FB15k datasets, and WN11 and FB13 datasets. In triple classification tasks, we used WN11 and FB13 datasets, as well as FB15k. Note that WN18 and FB15k are originally used for link prediction tasks, whereas WN11 and FB13 are originally used for triple classification tasks. Also note that WN18 cannot be used for the triple classification task because WN18 does not have test negative data. TAB4 shows the number of the entities, relations, and triples in each dataset. Manifolds in Riemannian TransE To evaluate the dependency of performance of Riemannian TransE, we compared Riemannian TransE using the following five kinds of manifolds: Euclidean space R D (Euclidean TransE), hyperbolic spaces H D (HyperbolicTransE), spheres S D (SphericalTransE), the direct product H 4 × H 4 × · · · × H 4 of hyperbolic spaces (PHyperbolicTransE), and the direct product S 4 × S 4 × · · · × S 4 of spheres (PSphericalTransE). We compared our method with baselines. As baselines, we used. We used implementations of the baselines in OpenKE http://openke.thunlp. org/static/index.html, a Python library of knowledge base embedding based on Tensorflow BID0, and moreover, we implemented some lacked constraints (for example, in TransR, TransD) and regularizers (for example, in DistMult, Analogy) in OpenKE. We also found that omitting the constraint of the entity planets onto sphere in TransE gives much better in our setting, and this is why we also show the without the constraint (UnconstraintTransE). We also implemented Riemannian TransEs as derivations of the base class of OpenKE.We set the dimensionality of the entity manifold as D = 8, 16, 32, 64, 128. Although we also have to determine the dimensionality of the projected space in TransR and TransD, we let them be equal to D. Due to limitation of the computational costs, we fixed the batch size in baselines and Riemannian TransEs such that that the training data are split to 100 batches. We also fixed the number of epochs to 1000. Note that in the first 100 epochs in Riemannian TransEs, we fixed the launchers. Also note that we applied norm clipping such that the norm of a stochastic gradient in the tangent space is smaller than 1. We did not use "bern" setting introduced in , where the ratio between head and tail corruption is not fixed to one to one; in other words, we replaced head and tail with equal probability. Other than the dimensionality and batch sizes, we used hyperparameters such as learning rate η and margin paremeter δ of baselines used in each paper. Note that some methods only reports link prediction tasks, and reports hyperparameters for WN18 and FB15k and do not reports ones for WN11 and FB13. Some methods do not mention settings of hyperparameters, and in these cases, we used the default parameters in OpenKE. In these cases, we used hyperparameters of WN18 and FB15k also for WN11 and FB13, respectively. Note that the parameters of TorusE is supposed to be used with very high dimensionality, and the hyperparameters are designed for high dimensionality settings. In Riemannian TransEs, we simply followed the hyperparameters in TransE.We used the Xavier initializer BID9 as an initializer. When we have to use the points on a sphere (in the original TransE and Spherical TransEs), we projected the points generated by the initialization onto the sphere. We found that choice of an initializer has significant effect on embedding performance, and the Xavier initializer achieves very good performance. We selected optimizers in baselines following each paper. Note that while using ADADELTA is also proposed in TransD, we used SGD in TransD. In Riemannian TransEs, we used we simply followed the hyperparameters in TransE. TAB5 shows the hyperparameters and optimization method for each method. BID2, it still contain many reversible triples, as noted by. By contrast, these are removed in WN11 and FB13. Recall that projection-based methods such as TransH, TransR and TransD, and inner-product-based methods such as ComplEx and DISTMULT can exploit a linear subspace. When a dataset has apparent clusters inside which one relation is easily recovered from the others, we can allocate each cluster to a subspace and separate subspaces from one another. This separation is easily realized by setting some elements in the launchers to zero in these methods. Indeed, the TransE without the sphere constraint attains good accuracies in WN11 and FB13.Differences between criteria are also interesting phenomena. Note that MRR and hit@10 is generous for heavy mistakes. It is possible that inner-product-based methods earn good scores in trivial relations, but further intensive investigation is needed.

Represent the short summary for finding the corresponding scientific article (long)

Multi-relational graph embedding with Riemannian manifolds and TransE-like loss function.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.