Towards Better Benchmark Datasets for Inductive Knowledge Graph Completion
Knowledge graphs;graphs;link prediction
Yes
No ethics review needed.
1. **Analysis of Other (E, R) Inductive Datasets**: As discussed in W1, PediaTypes and WikiTopics are two fully inductive cross-domain KG datasets proposed by [1] that were not studied in this paper. Could you consider applying the same analysis to these datasets to determine whether they also exhibit divergent SPD differences between positive and negative samples? Additionally, could you evaluate the performance of PPR on these datasets? Including results for these datasets would significantly enhance the paper's comprehensiveness, particularly regarding its coverage of (E, R) inductive datasets.

2. **Evaluation of Other (E, R) Inductive Methods on New Inductive Dataset**: As discussed in W1, Gao et al. [1] proposed the (E, R) inductive methods ISDEA+ and DEq-InGram. Including these methods in the baseline experiments for Section 5 would provide a more comprehensive comparison with the other methods tested in the paper. Of particular interest is that the paper notes that "InGram struggles in the (E, R) setting" on the newly created dataset based on graph partitioning; therefore, it would be valuable to determine whether DEq-InGram could overcome these struggles or if it still faces similar challenges.

3. **Question about the Inference Setting on the (E, R) Inductive Experiments**: It seems that the results of ULTRA [2] on the new inductive datasets are placed in Figure 6, separate from the main (E, R) results shown in Table 4. Could you clarify why these results are presented separately? Is the distinction due to different inference settings (e.g., zero-shot inference for ULTRA in Figure 6 versus few-shot fine-tuning or test-time adaptation for other methods in Table 4)? If this is the case, explicitly mentioning this distinction in the experimental setup section would help reduce confusion and improve clarity.

**References**

[1] Gao, Jianfei, et al. "Double equivariance for inductive link prediction for both new nodes and new relation types." (2023)

[2] Galkin, Mikhail, et al. "Towards foundation models for knowledge graph reasoning." (2023)

[3] Lee, Jaejun, Chanyoung Chung, and Joyce Jiyoung Whang. "InGram: Inductive knowledge graph embedding via relation graphs." (2023)
1. **Addressing a Fundamental Problem in Existing Inductive Datasets**: The paper identifies that the shortest path distance (SPD) shortcut is prevalent among several existing inductive KG datasets. Furthermore, the authors demonstrate that all datasets generated by varying configurations of the existing data generation algorithm suffer from this problem. This indicates that the current inductive data generation strategy is fundamentally flawed, and addressing these flaws is crucial to genuinely evaluate a model's inductive capabilities.

2. **Well-Written and Clear Presentation**: The paper is well written and easy to follow. The problem statement, methodology, and experimental findings are presented in a clear and structured manner.

3. **Effective Remedial Strategy**: The authors propose an intuitive and effective strategy for sampling inductive KGs based on graph partitioning, ensuring that entities within a partition are densely connected while inter-partition connections are minimized. This maintains the original graph's SPD distribution, mitigating shortcuts and providing a fairer benchmark.

4. **Extensive Empirical Study**: The authors conducted an extensive empirical evaluation, providing a thorough analysis of both the existing and newly proposed datasets. The experiments cover a wide range of inductive KG completion methods, adding significant value to the paper.
The main finding of the paper is that many existing inductive knowledge graph (KG) datasets are constructed in a way that creates significant differences in shortest path distances (SPD) between positive and negative samples. This discrepancy allows non-learning-based heuristics, like Personalized PageRank (PPR), which ignores relation types altogether and is unsuitable for knowledge graph completion (KGC), to achieve performance close to that of SOTA methods. This highlights that existing inductive KG datasets might be too simplistic, thereby failing to adequately test the full inductive capabilities of models.

The authors propose a novel dataset construction strategy using graph partitioning to ensure that the sampled subgraphs maintain the original KG structure while mitigating the introduction of SPD-based shortcuts. The proposed benchmark datasets are shown to be more challenging for inductive models, providing a better platform for evaluating their real capabilities.
**My main concern with this paper is that it misses key citations in Related Work**. Specifically, Gao et al. [1] was a concurrent work to Galkin et al. [2] that first provided a theoretical understanding of what is necessary for solving the (E, R) inductive KGC task. Gao et al. also introduced two new (E, R)-capable methods, ISDEA+ and DEq-InGram, with the latter being an improved version of the original InGram [3]. Additionally, two new (E, R) datasets, PediaTypes and WikiTopics, were proposed for empirical evaluation. Including discussions of all (E, R) capable methods, i.e. InGram (DEq-InGram), ULTRA, ISDEA+, in Section 2 (around lines 134-139) as well as including these datasets and methods in the empirical results would provide a more complete overview of recent advances in inductive KGC research and support a more comprehensive empirical comparison.
Yes
No ethics review needed.
Older datasets have simple patterns than new datasets. Why performance of NBFNet improves on new datasets while these datasets are more challenging?

Why the performance difference of PPR and NBFNet on WN18RR dataset in both old and new versions are the same, but this is not the case for FB237?
-- Numerous machine learning and AI approaches have been introduced in the literature. To demonstrate the effectiveness of new models compared to older ones, several standard datasets have been established for evaluation and conclusion. However, issues with these datasets can lead to misleading conclusions within the community about model performance. Therefore, the paper's original motivation and the problem it addresses are both significant and valuable.

– Overall, the paper is written well and easy to follow and highlight different aspects. 

– The results are consistent with what claimed, i.e., the performance of existing models on new inductive datasets are lower than older ones and also PPR obtain lower results on new datasets.
Inductive link prediction aims to predict missing links in test sets where entities, relations, or both have not been encountered during training. Numerous datasets have been developed as benchmarks for evaluating inductive link prediction methods. This paper identifies a flaw in existing inductive link prediction datasets: a simple, non-trainable method called personalized PageRank (PPR) can achieve performance comparable to complex machine learning approaches. This is largely because current datasets are structured such that the shortest path distance (SPD) between entities in positive test samples is considerably shorter than in negative samples. Consequently, a non-trainable algorithm like PPR, which disregards relation information, can still rank positive samples above negative ones, yielding high performance.
To address this issue, the authors propose new datasets for inductive link prediction, carefully partitioned from transductive datasets to prevent the PPR shortcut. The new benchmark reveals that models perform worse on these updated datasets compared to older ones, highlighting the need for more robust evaluation standards in inductive link prediction.
– Some part of the paper requires a better and more detailed explanation, e.g., the description related to page rank which is important to better understand the issue.

– that the deltaSPD of train and inference should be close is understandable. But it is not very clear why this partitioning solve the main raised issue. 

–  the analysis of the tables should be more detailed and cover interpretation of various observations.
Yes
No ethics review needed.
Please refer to Weaknesses.
1. This work provides a clearer understanding of the capabilities and challenges in inductive KGC and the benchmark construction.

2. The manuscript is well-organized and easy to follow.

3. Experiments have verified the effectiveness of the proposed benchmarks.
This paper identifies a shortcut in current inductive KGC datasets, where simple methods like Personalized PageRank can achieve strong performance without using relational information. The authors propose a new dataset construction strategy to eliminate this shortcut and benchmark popular KGC methods on these datasets.
1. The technical contribution is kind of limited for a long paper. The applied graph partitioning strategy is straightforward and the research challenge in this paper is not significant.

2. There is a confusing point in the benchmark construction. Why the differences in SPD are beneficial from graph partitioning sampling? Since “the different in mean SPD” (also a typo in line 237) causes this shortcut, why not improve the negative sampling strategy directly? For example, select entities having a much shorter distance from the query entity as negative samples. It could be independent of the graph structure.

3. In my opinion, the shortcut proposed in this paper is mainly caused by the negative-sampled evaluation (distinguishing the positive entity from a fixed number of negative ones). Recent studies, including RED-GNN and NBFNet, already employ the full evaluation (finding the positive entity from the entire entity set). This might be the reason for the superior performance of the two models as well as ULTRA. From this point, have the authors compared the performance differences between the two evaluation settings on new benchmarks?
Yes
No ethics review needed.
1. Is it possible to evaluate some cases of new drugs, new patients, and cold start recommendations? 

2. Whether phenomena like the correlation between PPR and SPD exist in these applications?

3. How to guarantee the sets of entities or relations are different by graph partitioning?
1. The logic in this paper is smooth and easy to capture.
2. The analysis of the relationship between PPR, SDP, and KGC performance is studied in detail.
3. The results of the split generated by the graph partition method are consistent with the detailed analysis.
This paper studies the datasets for inductive KGC. First, it is observed that PPR can have hear SOTA performance on most inductive datasets. Then, some analytical experiments are conducted to study the correlation between PPR and shortest path distance. Based on the observations, the authors propose a new split method through graph participation algorithms.
1. Even though the evaluation of PPR and SPD on current inductive benchmarks reals some problems and sounds reasonable, the match of these problems with realistic demands is not fully discussed. In particular, there lack of discussion and evaluation of datasets in real-world inductive settings. At the beginning of the introduction, KGC applications include drug discovery, personalized medicine, and recommendations. Is it possible to evaluate some cases of new drugs, new patients, and cold start recommendations? I want to know whether phenomena like the correlation between PPR and SPD exist in these applications.

2. Similar to problem 1, I think the main purpose of recreating the inductive dataset is to match it with its transductive counterparts. In my mind, the transductive datasets also have unrealistic problems.

3. Some details of the graph partition method in data creation are not clear. In particular, I care about how the graph partition method can guarantee that the training and inference datasets have different sets of entities or relations. Maybe I missed some important content, but I failed to find this point in both the main content and the appendix.

4. Another commonly used inductive dataset NELL is not studied in this paper.
@inproceedings{
anonymous2024towards,
title={Towards Better Benchmark Datasets for Inductive Knowledge Graph Completion},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=npBAHV5BJI},
note={under review}
}
Knowledge Graph Completion (KGC) attempts to predict missing facts in a Knowledge Graph (KG). Recently, there's been an increased focus on designing KGC methods that can excel in the *inductive setting*, where a portion or all of the entities and relations seen in inference are unobserved during training. Numerous benchmark datasets have been proposed for inductive KGC, all of which are subsets of existing KGs used for transductive KGC. However, we find that the current procedure for constructing inductive KGC datasets inadvertently creates a shortcut that can be exploited even while disregarding the relational information. Specifically, we observe that the Personalized PageRank (PPR) score can achieve strong or near SOTA performance on most inductive datasets. In this paper, we study the root cause of this problem. Using these insights, we propose an alternative strategy for constructing inductive KGC datasets that helps mitigate the PPR shortcut. We then benchmark multiple popular methods using the newly constructed datasets and analyze their performance. The new benchmark datasets help promote a better understanding of the capabilities and challenges of inductive KGC by removing any shortcuts that obfuscate performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Knowledge graphs graphs link prediction
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9b833cf12b17da7f3a3bb4ec0e9ecc3c6d4cbe85.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Towards Better Benchmark Datasets for Inductive Knowledge Graph Completion
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission