Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval
EHR Prediction;Large Language Models;Knowledge Graphs
Yes
No ethics review needed.
1. In Section 3.1.1, three different KGs are generated for each medical concept, and the final KG is to integrate the three KGs together. How do you handle the conflicts among the three KGs?
1. The proposed method improves the interpretability and trustworthiness of clinical predictions by generating reasoning chains.
2. The experiments are extensive. The framework is compared to many baselines and shows a sufficient performance gain in both tasks.
3. The visualizations are well-structured and the writing is easy to follow.
4. The problem setting is clearly defined.
The paper presents KARE, a framework designed to improve healthcare predictions by combining KG community-level retrieval with LLM reasoning. It builds a medical KG from diverse sources, organizes it into meaningful communities, and dynamically augments patient data with relevant information. Extensive tests on MIMIC-III and MIMIC-IV datasets demonstrate significant performance improvements for mortality and readmission prediction.
1. The framework is overly complex, making it difficult to optimize and reproduce. Moreover, any noise introduced in the earlier steps (e.g., entity and relation extraction, LLMs for relationship suggestions and reasoning chains generation, etc.) could affect subsequent stages and degrade the final performance.
2. Given its complexity, the framework's efficiency should be evaluated. It involves building three different KGs for each medical concept, conducting community detection 25 times, and generating multiple summaries for each community (~60k communities in total). This process likely requires significant time and resources, making it inefficient.
3. The contribution is a bit limited compared to GraphRAG; the primary difference is mentioned in line 213 that they run GraphRAG multiple times for better diversity.
4. Acronyms like LLM, KG, EHR, and RAG are introduced multiple times throughout the paper.
5. It would be better to include case studies for the three KGs generated by the biomedical KG, biomedical corpus, and LLMs.
Yes
No ethics review needed.
please refer to the weaknesses
1. I really enjoyed the LLM-Driven Reasoning, producing both accurate and interpretable clinical predictions that enhance trust.
2. The model's multi-source knowledge graph ensures relevant information retrieval, addressing LLM limitations like hallucinations and sparse data.
3. KARE's retrieval mechanism enriches patient data with multi-faceted insights, enhancing EHR representation learning
The paper presents KARE, a framework that enhances healthcare predictions by combining knowledge graph (KG) community-level retrieval with large language model (LLM) reasoning. It addresses LLM limitations like hallucinations and inadequate medical knowledge, which can affect clinical diagnosis. KARE builds a comprehensive knowledge graph from biomedical databases, clinical literature, and LLM insights, organized through hierarchical community detection and summarization to improve retrieval precision and relevance. Key innovations include: (1) a dense medical knowledge structuring approach for accurate information retrieval; (2) a dynamic retrieval mechanism that enriches patient contexts with multi-faceted insights; and (3) a reasoning-enhanced prediction framework that produces accurate and interpretable clinical predictions.
1. Methodology Clarity: Some aspects of the methodology lack clarity, such as how the different knowledge graphs (KGs) are connected in Equation (1). Specifically, the approach for creating edges between nodes in different KGs, like G^KG and G^BC, is not well explained.

2.  The experimentation could be broadened to include more general tasks, such as diagnosis prediction or drug recommendation. Was there a reason these broader tasks were not considered?

3.  The results lack standard deviations or confidence intervals, which would help indicate the reliability of the reported performance.

4. Ablation Study Design: The ablation study could be more informative if it involved removing each feature individually, rather than adding features one at a time.
5. The provided anonymous GitHub code link cannot be opened.
Yes
No ethics review needed.
See Weaknesses Above.
- Pros
  - Interesting topic for enhancing healthcare predictions by combining the reasoning capabilities of LLM.
  - Clear and well-motivated reasoning in the paper. 
  - Comprehensive ablation studies validate the contributions of each model component.
  - Well-written and structured.
This paper introduces KARE, a framework integrating LLM reasoning with KG retrieval to improve healthcare predictions. KARE combines structured multi-source medical knowledge with dynamic, patient-specific context augmentation to provide accurate and interpretable clinical predictions. Evaluated on MIMIC-III and MIMIC-IV datasets for mortality and readmission predictions, KARE demonstrates improved accuracy and interpretability over conventional models.
- Cons
  - The novelty of this paper is relatively limited, appearing to be incremental compared to GraphRAG [1].
  - In Section 3.3.1, the authors select the reasoning chain with the highest confidence as training data. However, according to conclusions from some existing studies [2,3], the reasoning chain with the highest confidence is not necessarily the most reliable.
  - MedRetriever [4] also adopts a retrieval-augmented approach for healthcare prediction, but this paper lacks a comparative analysis with MedRetriever.
  - The definitions and calculation methods for Sensitivity and Specificity need to be clarified more thoroughly, and metrics such as AUROC and AUPRC should be added.
  - Although Amazon Bedrock provides strict compliance standards and privacy protection measures, relying on it to generate reasoning chains for distillation may limit the generalizability of this approach in real healthcare scenarios with high privacy protection requirements.

[1] Edge D, Trinh H, Cheng N, Bradley J, Chao A, Mody A, Truitt S, Larson J. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. 2024 Apr 24.

[2] Yang, Haoyan, et al. "Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfer." arXiv preprint arXiv:2405.16856 (2024).

[3] Xiong, M., Hu, Z., Lu, X., LI, Y., Fu, J., He, J. and Hooi, B., Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. In The Twelfth International Conference on Learning Representations.

[4] Ye M, Cui S, Wang Y, Luo J, Xiao C, Ma F. Medretriever: Target-driven interpretable health risk prediction via retrieving unstructured medical text. InProceedings of the 30th ACM International Conference on Information & Knowledge Management 2021 Oct 26 (pp. 2414-2423).
Yes
No ethics concerns. To my knowledge, EHR data is fuzzified, for example, the patient's visit time information is also time offset. And the author also used local LLM, so I don't think there are any ethical concerns. The author has already provided details in Appendix A.
No ethics review needed.
1. The authors used Claude 3.5 Sonnet as an expert model to generate training samples and augment knowledge graph. However, since Claude is a general-purpose model, could it lack some medical knowledge, potentially leading to biased training samples and cumulative errors? As mentioned in your summary: "Yet LLMs still suffer from hallucinations and lack fine-grained contextual medical knowledge, limiting their high-stakes healthcare applications such as clinical diagnosis." In the experiment section, there are many related LLMs in the medical domain. It would be better if the researcher could compare KARE with more related LLM-based baselines referred in [4].
2. I didn't see any examples of training samples in the code you provided. Can you provide us with some examples?
3. This parameter design is very challenging. In KARE, there are many hyperparameters, including but not limited to those in graph generation, summarization, model training, and model testing. Any slight change can lead to significant deviations in the model's results. Could you elaborate on how you adjust the parameters in such a large hyper-parameter space?


- [4] https://huggingface.co/blog/leaderboard-medicalllm.
1. The research team has achieved impressive results, exceeding established benchmarks. This progress marks a significant step forward in improving predictive model accuracy in medical data analysis.
2. The authors' workload is immense, and the experimental details are thoroughly outlined, which I greatly appreciate.
The paper introduces KARE, a novel framework designed to enhance clinical decision support by addressing the limitations of Large Language Models (LLMs) in healthcare. While LLMs show potential, they suffer from hallucinations and lack the fine-grained medical knowledge necessary for high-stakes applications like diagnosis. Traditional retrieval-augmented generation (RAG) methods often retrieve sparse or irrelevant data, undermining accuracy. KARE improves upon this by integrating a multi-source knowledge graph (KG) with LLM reasoning. The KG is built from biomedical databases, clinical literature, and LLM-generated insights, structured using hierarchical community detection for precise information retrieval. Key innovations include dense medical knowledge structuring, dynamic retrieval of multi-faceted medical insights, and reasoning-enhanced predictions. KARE outperforms existing models in MIMIC-III and MIMIC-IV datasets, improving prediction accuracy by up to 15%, while also enhancing the interpretability and trustworthiness of clinical predictions.
1. By reviewing your code and the details in the article, I can see that your workload is immense, however, the contribution of this article is incremental. My understanding is that it is essentially a combination of GraphRAG and GraphCare [1]. Furthermore, many key baselines were not cited. Since the authors mentioned that this paper focuses on RAG for EHR, some essential RAG algorithms should have been introduced, such as MedRetriever [2], and commonly used GraphRAG algorithms like KGRAG [3]. 
2. In the experiment or appendix section, I did not clearly see the formulas for Sensitivity and Specificity, nor were there any corresponding references, which is quite confusing to me. Moreover, using Accuracy as a metric in cases of highly imbalanced labels is unreasonable. For instance, in the MIMIC-III Mortality Prediction task, the positive rate is 5.42%. If I predict that all patients will survive, I can still achieve an accuracy of 94.58%. Previous works, such as GraphCare [1], have adopted AUROC and AUPRC as evaluation metrics.
3. The article is overly long and filled with detailed content, making it easy for readers to miss important points.

- [1] GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs. ICLR 2024
- [2] MedRetriever: Target-driven interpretable health risk prediction via retrieving unstructured medical text. CIKM 2021
- [3] Biomedical knowledge graph-enhanced prompt generation for large language models. Arxiv 2023
Yes
No ethics review needed.
* In Section 3.2, how does the method handle cases where the same patient has more than two visits with different labels? Is there a specific design to distinguish similar symptoms in different patients and multiple visits from the same patient?
* In Section 3.2, how is the effectiveness of the relevance score (i.e., Formula (3)) validated? Would an ablation study on the components of Relevance(C_k) help to clarify this?
* How does the method utilize patient longitudinal visit information?
* For the baseline ML methods, it’s mentioned that most are implemented using PyHealth. Could you clarify the backbone model for each ML method? Also, is there a fair configuration in place for implementing language model-based encoders, such as ClinicalBERT, across these methods? 
* The experimental results for mortality prediction show that the baseline ML methods, such as ConCare and TCN, perform closely to KARE, with some evaluation metrics even exceeding KARE’s. How should these results be interpreted?
* In experiment, sensitivity and specificity are indeed important metrics, but F1 score and accuracy are also widely used for assessing diagnostic accuracy. Interestingly, in ablation study, excluding similar patients often yields better or comparable results than including them, especially for mortality prediction on the MIMIC-III dataset. Do these results support the methodological choices made in Section 3.2?
* As you state that your base model is fine-tuned on Mistral-7B-Instruct-v0.3, I wonder about the performance of fine-tuning an LLM (Mistral or Llama) with RAG. Can your model still show substantial improvements compared to other LLM-based models beyond just zero-shot or few-shot settings? I am concerned that most of the current improvements may be attributed primarily to supervised fine-tuning.
* The paper is well-written and structured.
* The authors propose a novel knowledge-graph learning framework combined with LLMs.
* The integration of knowledge augmentation and knowledge graphs is both novel and interesting.
* The experimental evaluation is thorough, conducted on both the MIMIC-III and MIMIC-IV datasets.
In this manuscript, the authors propose KARE, a knowledge graph-enhanced large language model designed for predicting patient mortality and readmission, with an added aim to mitigate LLM hallucinations. The paper introduces a novel approach that effectively combines knowledge graphs with LLMs through a clustering method. Additionally, the authors present a reasoning-chain mechanism to enhance the LLM's inference capabilities and provide interpretable prediction results. Experimental results demonstrate that KARE achieves a marginal improvement in mortality and readmission predictions on the MIMIC-III and MIMIC-IV datasets. However, several major and minor issues need to be addressed. If author could well address my concern, I would like to update my ratings,
* The excessive use of symbols makes certain sections difficult to follow.
* The paper lacks a detailed description of longitudinal EHR processing with LLMs.
* The improvement in mortality prediction performance achieved by KARE is marginal.
* There is a lack of sensitivity analysis for hyperparameters, such as the number of top co-occurring concepts, top documents, and the criteria for selecting optimal values. Additional hyperparameters, such as maximum sequence length and maximum path count, should also be discussed, or at least identified explicitly for clarity.
* The paper would benefit from a more in-depth interpretability analysis to clarify the relationship between the generated reasoning, labels, and the knowledge graph.
* The paper lacks an evaluation of LLM hallucinations, despite the stated aim to address hallucination issues in clinical decision-making.
* An analysis of model parameters and time consumption is missing, which could provide valuable insights into the model’s computational efficiency and practical applicability
* There is a lack of references to related work.

  [1] . Kang, M., Lee, S., Baek, J., Kawaguchi, K., & Hwang, S. J. (2024). Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks. Advances in Neural Information Processing Systems, 36. 

  [2]. Niu, S., Ma, J., Bai, L., Wang, Z., Guo, L., & Yang, X. (2024). EHR-KnowGen: Knowledge-enhanced multimodal learning for disease diagnosis generation. Information Fusion, 102, 102069.

  [3]. Kwon, T., Ong, K. T. I., Kang, D., Moon, S., Lee, J. R., Hwang, D., ... & Yeo, J. (2024, March). Large language models are clinical reasoners: Reasoning-aware diagnosis framework with prompt-generated rationales. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 16, pp. 18417-18425).
@inproceedings{
anonymous2024reasoningenhanced,
title={Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8fLgt7PQza},
note={under review}
}
Large language models (LLMs) have demonstrated significant potential in clinical decision support. Yet LLMs still suffer from hallucinations and lack fine-grained contextual medical knowledge, limiting their high-stake healthcare applications such as clinical diagnosis. Traditional retrieval-augmented generation (RAG) methods attempt to address these limitations but frequently retrieve sparse or irrelevant information, undermining prediction accuracy. We introduce KARE, a novel framework that integrates knowledge graph (KG) community-level retrieval with LLM reasoning to enhance healthcare predictions. KARE constructs a comprehensive multi-source KG by integrating biomedical databases, clinical literature, and LLM-generated insights, and organizes it using hierarchical graph community detection and summarization for precise and contextually relevant information retrieval. Our key innovations include: (1) a dense medical knowledge structuring approach enabling accurate retrieval of relevant information; (2) a dynamic knowledge retrieval mechanism that enriches patient contexts with focused, multi-faceted medical insights; and (3) a reasoning-enhanced prediction framework that leverages these enriched contexts to produce both accurate and interpretable clinical predictions. Extensive experiments demonstrate that KARE outperforms leading models by up to 10.8-15.0\% on MIMIC-III and 12.6-12.7\% on MIMIC-IV for mortality and readmission predictions. In addition to its impressive prediction accuracy, our framework leverages the reasoning capabilities of LLMs, enhancing the trustworthiness of clinical predictions.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
EHR Prediction Large Language Models Knowledge Graphs
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a7183dca9fd82593f6080cf05c99d07e88aab0bd.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/16b031a842f4dcdbc45cec41a7706ce649093375.zip
Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission