Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models
Large Language Models;Medication Consultation;Retrieval-Augmented Generation;Tool Call
Yes
No ethics review needed.
Typo? The manuscript mentioned "Table 3.2," but this is not Table 3.2; I guess it should be Table 1.
Question: Will the medicine database be released to the public?
This paper proposed a new dataset for medication consultation.
This paper built a multiple-round dialogue benchmark MedicineQA and proposed a distill-retrieve-read framework for RAG improved medication consultation capability of LLM. The dataset size is limited, and the multiple-round dialogues were generated by GPT-4 but not from real-world dialogue; this may affect the value of the application for the proposed framework. More details about the dataset should be introduced, such as the number of dialogue rounds.  It seems the dialogue distilling tool was trained on the general domain synthetic dataset but not for the medical domain, and the details of the dialogue distill tool training are not clear (which base model? data size? training settings? list of data sources). The proposed method lacks novelty; it applies the dialogue distilling module in the RAG setting and tests on the small MedicineQA dataset. Experiments are only based on a single dataset.
Typo? The manuscript mentioned "Table 3.2," but this is not Table 3.2; I guess it should be Table 1.
Question: Will the medicine database be released to the public?
The dataset size is limited, and the multiple-round dialogues were generated by GPT-4 but not from real-world dialogue; this may affect the value of the application for the proposed framework. More details about the dataset should be introduced, such as the number of dialogue rounds.  It seems the dialogue distilling tool was trained on the general domain synthetic dataset but not for the medical domain, and the details of the dialogue distill tool training are not clear (which base model? data size? training settings? list of data sources). The proposed method lacks novelty; it applies the dialogue distilling module in the RAG setting and tests on the small MedicineQA dataset.  Experiments are only based on a single dataset.
Yes
No ethics review needed.
1) What is the use of the question annotations in MedicineQA? If I understand correctly, it is the explicit reformulation of the questions from the conversation history (which might be ambiguous and contain superfluous information, as shown in "Dialogue History" in Figure 3). I understand that the synthetic dataset collected and described in lines 257-264 is used to convert questions into keywords, but during inference, what is actually used as input? Is it the dialogue history, of the extracted questions. If the latter, how are they extracted from dialogue history?

2) Could you please indicate how you measure the H@K for "baseline models". From reading lines 300-306, It is not clear to me what the baselines are actually doing. Are they generating the medication names from their parameters? Are they generating queries that are run against the database? 

3) Furthermore, could you please elaborate on what you mean by "we immediately adopt the attached corresponding medicine information as the context to guide the generation [...]" in line 323. Is it the ground-truth medicine annotations from MedicineQA? If so, I find this result highly counter-intuitive and contradictive, as the paper states later, in lines 389 ff, that the improvements of RagPULSE are due to the proposed RAG framework rather than due to domain fine-tuning of the PULSE LLM. But since RagPULSE's retrieval is far from optimal (e.g. H@k rates in Tables 2 and 3), it would suggest that imperfect retrieval somehow leads to better generations. Similarly, could you please indicate what "instruction used for baseline models" in the description of Table 3 means: "We also prompt RagPULSE by the instruction used for baseline models, which are denoted as †".
The collected resources seem helpful and, even if collected automatically, are validated by domain experts. The proposed approach seems to improve over LLM-based baselines.

The experiments seem sound in what they report/investigate.

The overall writing quality is good.
This paper introduces a benchmark, database and method for describing medications in dialogues with LLMs. The benchmark is collected automatically and reviewed by medical experts and the database is curated to support retrieval of medication information. The paper evaluates the proposed approach of converting conversations with their histories into a set of keywords to query the medical database and the results indicate favourable outcomes, based on GPT-4- and human-based evaluations of the generations.
My main concerns is the lack of meaningful comparison and under-specification of some of the details.

- While the paper suggests that the proposed distill-then-retrieve framework is better than RAG methods, I did not find meaningful comparisons with either general RAG frameworks as e.g. discussed in lines 117-126, or with domain-specific RAG approaches, such as e.g. BioReader (https://aclanthology.org/2022.emnlp-main.390.pdf) RAFT (https://arxiv.org/abs/2403.10131).

- The experimental setting is somewhat under-specified, see my questions to the authors. In summary, I don't understand the exact settings for the comparisons in Tables 2 and 3.

- There is very little detail on the construction of the medical database, which seems to be the key contribution enabling the retrieve-then-distill framework. Particularly, data sources, data quality measures etc are missing.

I gravitate towards rejection but if my queries are answered with sufficient detail, I am open to re-evaluating my decision.
Yes
No ethics review needed.
1. How ragpulse deals with retrieval errors (if any)? Also, did you see any failure modes commonly found and can you discuss them, please?

2. The benchmark is available in English and Chinese as per the paper. but how well do we know if MedicineQA can be adapted to other languages (maybe low-resource languages)?
This paper shows an approach to enhancing LLMs in the medical domain by reformulating rag for multi-round medication consultations. Also, using a tool-calling mechanism for search query formulation is a novel method of tailoring retrieval processes in dialogue-based tasks in healthcare. The paper shows good quality in methodology. Also, the authors did around 600 experiments to validate the effectiveness of ragpulse and provided comparisons with existing open-source and commercial models. The paper has clear objectives, methods, and findings sections written. The study addresses a research gap in applying LLMs to the medical field and introduced MedicineQA, ragpulse. Their work shows good retrieval and response generation capabilities.
The paper has introduced a benchmark called 'MedicineQA' to evaluate llms in medical consultation. Their suggested benchmark is focusing on rag for multi-turn dialogues in medication consultation. The authors also propose RagPULSE framework to deal with limitations of standard RAG frameworks. This framework utilizes a tool-calling mechanism to enhance retrieval accuracy by summarizing dialogue history into search queries and retrieving evidence from a medicine database.
I feel that the number of multi-round dialogues (300) are very less and that makes it difficult in terms of generalizing the results. So, maybe increase the dataset size by expanding the set of topics like for various specific conditions like breast cancer, diabetes, chronic pain.

Another thing is that the study should discuss retrieval errors for the cases where the model fails to retrieve accurate responses. The authors can try to include 'error analysis' section (maybe) that show various retrieval failures types or examples. This analysis will certainly go a long way in understanding if any adjustments should be made in distillation process.
This study introduces a benchmark containing multi-round dialogues for medication consultation and a novel retrieval-augmented framework with the tool calling mechanism.
@inproceedings{
anonymous2024tool,
title={Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=jtY2OHKj4a},
note={under review}
}
Large-scale language models (LLMs) have achieved remarkable success across various language tasks but suffer from hallucinations and temporal misalignment. To mitigate these shortcomings, Retrieval-augmented generation (RAG) has been utilized to provide external knowledge to facilitate the answer generation. However, applying such models to the medical domain faces several challenges due to the lack of domain-specific knowledge and the intricacy of real-world scenarios.
In this study, we explore LLMs with RAG framework for knowledge-intensive tasks in the medical field. To evaluate the capabilities of LLMs, we introduce MedicineQA, a multi-round dialogue benchmark that simulates the real-world medication consultation scenario and requires LLMs to answer with retrieved evidence from the medicine database. MedicineQA contains 300 multi-round question-answering pairs, each embedded within a detailed dialogue history, highlighting the challenge posed by this knowledge-intensive task to current LLMs. We further propose a new \textit{Distill-Retrieve-Read} framework instead of the previous \textit{Retrieve-then-Read}. Specifically, the distillation and retrieval process utilizes a tool calling mechanism to formulate search queries that emulate the keyword-based inquiries used by search engines. With experimental results, we show that our framework brings notable performance improvements and surpasses the previous counterparts in the evidence retrieval process in terms of evidence retrieval accuracy. This advancement underscores the framework's potential to effectively address the inherent challenges of applying RAG models to the medical domain.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Medication Consultation Retrieval-Augmented Generation Tool Call
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1bd458972eea38f0ff8356a6fdd1ba17e812c167.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission