EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers
Biomedical Benchmark;Scientific Information Retrieval;Scientific Information Extraction;Large Language Models;BioNLP
Yes
No ethics review needed.
Can you provide more insights or knowledge that we can learn?
1. Identifying arguments supporting or against an argument is an important task.
2. This paper proposes an annotation pipeline, which might be helpful for future tasks.
This paper presents EvidenceBench, which is a benchmark for finding arguments supporting or against a hypothesis. They find hypotheses inside survey, and find supporting arguments from the same survey, and the original paper. 
This paper also proposes metrics for the task and compares many LLMs' performance on the task. In addition, this paper also provides the fine-tuning results on the benchmark.
1. The paper spends too much effort on constructing the benchmark, but not many insights are provided through the experiment section.
2. The writing can be largely improved. There's many places in the writing that are vague and not clear. 
For example, the second paragraph in the introduction section: 
"We consider the goal of understanding what is known in the literature about a scientific hypothesis.
This can be broken into several stages: searching for relevant papers; extracting information from
these papers; and aggregating this information. Our work focuses on the second stage."
My questions are:
a). what is precisely the research goal in terms of "understanding what is known in the literature about a scientific hypothesis"?
b). why it can be broken into these three stages?

In line 047~048: "These annotations are judgments which are ordinarily made by domain experts, and the benchmark should be faithful to these judgments": what does it mean by "the benchmark should be faithful to these judgements"?

In line 048~050: "Third, despite the complexity of annotation, the benchmark construction process should be scalable, providing a sufficient number of examples to accurately measure system performance": what does it mean by "should be scalable"? what does it mean by "providing a sufficient number of examples to accurately measure system performance"? what is the relation between these two arguments?

I found it very hard to read through the paper. I would suggest the authors for a major revision at least in terms of the writing.
Yes
No ethics review needed.
N/A
* The authors propose a practical task and evaluate the effectiveness of existing approaches to the task
* A useful resource for benchmarking LLMs on the proposed task
* The paper is well-structured and easy to follow, although some presentations can be further improved
This paper studies the task of finding evidence for a hypothesis. The authors develop a pipeline for annotating biomedical papers for this task. Using the annotation pipeline, the authors build a benchmark of more than 400 papers. Additionally, the authors create a larger dataset containing 100K papers.
The authors also run experiments to evaluate the effectiveness of different approaches to the proposed task.
There is no strong reason to reject the paper, although some issues related to clarity and presentation need to be improved.

* The concept of `study aspect` is confusing, and I am unsure how the evaluation procedure considers it. For example, if a sentence is retrieved for the wrong aspect or multiple aspects.
* Based on the task definition, the hypothesis is given. However, this may not be the case in the real world. It would be nice to see these tested models' sensitivity to the modified (paraphrasing) hypothesis.
* The difference between `EvidenceBench` and `EvidenceBench-100k` is unclear (seems with or without human validation?)
* The authors exclude figure and table, which might be very relevant and important for the proposed task.
Yes
This paper released a new dataset for evidence extraction from biomedical papers, which requires some ethic evaluations of the dataset creation.
Yes, Responsible research practice (e.g., human subjects, data release)
o	How is the 3 human hour for construction using this pipeline concluded? Why are different LLMs (GPT4-0125, GPT4-o-mini, and Claude3-Opus) selected as the tool at different stage of benchmark data creation?

o	The summary extraction from study monographs is performed by LLMs, without human inspection. The summaries are used as input to LLMs for the recovery of hypotheses and decomposition of aspects. Although the aspect decomposition is inspected by humans, how is the summary extraction validated to avoid error propagation?

o	In Task Definition, both versions of the task define the desired sentences as evidence for or against a hypothesis. It is difficult to discern whether sentences classified as counter-evidence in relation to one hypothesis might be more appropriately considered as supportive of an alternative hypothesis. Given the structured nature of the tasks, are there any experiments or human validations conducted to address and clarify these categorizations?
o	The original EvidenceBench evaluated the performance of selected LLMs and embedding models on evidence retrieval tasks and compared different prompting strategies, which revealed that GPT-4o is the SOTA LLM, and that embedding models underperform due to a lack of context awareness.

o	The EvidenceBench-100k fine-tuned E5-v2 model and Llama3-8B significantly improved on the result evidence retrieval task but trailed behind larger models, validating the effectiveness of the benchmark dataset.

o	The author presented the topic and their framework well, with detailed descriptions and clear figures illustrating the overall problem and their area.

o	This paper innovatively developed the pipeline for evidence retrieval for a given hypothesis and further annotated biomedical papers at the sentence level for better meta-analysis.

o	The authors conducted comprehensive experiments with both open-sourced and closed-sourced LLMs, and a small language model. Human experts were involved to validate study aspects generation and automate sentence annotation to enhance trustworthiness.
This paper introduces EvidenceBench, a scalable annotation pipeline designed for extracting and aligning evidence with specific hypotheses in biomedical literature. Study aspects and hypotheses are initially extracted from systematic reviews. A specialized alignment annotator then performs sentence-level annotations to link each piece of evidence directly to the corresponding hypothesis. To validate this approach, the authors use the pipeline to generate the expansive EvidenceBench-100k benchmark. Fine-tuned on this benchmark dataset, embedding models showed improved performance in the 'Result ER@Optimal' task, showing this standardized benchmark and evaluation framework will support the development of tools for automate evidence synthesis and hypothesis testing.
o	The authors claimed that their method using the SOTA LLMs reduces construction time from over 3,000 human hours for EvidenceBench to 3 hours. However, there is no evidence provided regarding how the 3 hours were concluded. Additionally, GPT4-0125, GPT4-o-mini, and Claude3-Opus are used during data generation without explanations of when to choose which.

o	No ablation study regarding how topics can influence the study aspect and hypothesis extraction is provided. The subsets used for experiments are randomly selected without considering the distribution of topics.

o	The experiments are conducted on a subset of the test dataset. Although EventBench-100K is comprehensive and large, only 300 data points are used to evaluate LLMs, which is a very small portion.
Yes
No ethics review needed.
- Hypotheses are taken from survey monographs, so they can differ from natural hypotheses the experts consider. Is there any evaluation of the dataset that evaluates the suitableness of the instances for evidence retrieval?
- What happens if the experts think of the hypotheses with no evidence?
- The large-scale datasets are constructed using LLMs with fewer budgets.
- Several evaluations of existing and fine-tuned models are provided and compared, showing the usefulness of the benchmark dataset for evaluating current LLMs' abilities.
This paper proposes a new task of finding evidence for a hypothesis. The authors built a large-scale dataset with reasonable costs using existing survey monographs and LLMs.
- It needs to be clarified how the hypotheses from survey monographs are generally helpful.
- The authors expect to provide immediate value to scientists as the first desiderate of the benchmark (lines 44-45), and the dataset creation involves several experts. Still, no manual analyses are provided for the results, so whether the results benefit scientists is unclear. 
- The proposed task focuses on the limited part of the practical problem; the task expects the candidate pool and needs to consider cases with evidence and with (the retrieval results of) the considerable paper pool (e.g., the entire PubMed database).

Typo:
- Line 509, Figure 3 should be Table 6.
A large-scale BioNLP benchmark for evaluating and training models on the task of Evidence Retrieval for hypotheses
@inproceedings{
anonymous2024evidencebench,
title={EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=NPDnRLFhc0},
note={under review}
}
We study the task of finding evidence for a hypothesis in the biomedical literature. Finding relevant evidence is a necessary precursor for evaluating the validity of scientific hypotheses, and for applications such as automated meta-analyses and scientifically grounded question-answering systems. We develop a pipeline for high quality, sentence-by-sentence annotation of biomedical papers for this task. The pipeline leverages expert judgments of scientific relevance, and is validated using teams of human annotators. We evaluate a diverse set of language models and retrieval systems on the benchmark, which consists of more than 400 fully annotated papers and 700k sentence judgments. The performance of the best models still falls significantly short of expert-level on this task. To show the scalability of our annotation and hypothesis generation pipeline, we created a larger EvidenceBench-100K containing 100,000 fully annotated papers with hypotheses and 150 million sentence judgments. Empirically, we show that fine-tuning embedding and language models on EvidenceBench-100k significantly improved their performance on the original EvidenceBench, reaching state-of-the-art embedding model performance and closely trails behind Large Language Models. By providing a standardized benchmark and evaluation framework, this work will support the development of tools which automate evidence synthesis and hypothesis testing, as well as the long-context global reasoning and instruction-following abilities for Large Language Models (LLM) and embedding-based IR systems. The dataset is available at \href{https://github.com/EvidenceBench/EvidenceBench}{https://github.com/EvidenceBench/EvidenceBench}.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Biomedical Benchmark Scientific Information Retrieval Scientific Information Extraction Large Language Models BioNLP
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/802036ca32606036aaec7d703a6c7d9fd47f714f.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission