Tooling or Not Tooling? The Impact of Tools on Language Agents for Chemistry Problem Solving
Chemistry;LLMs;Agent;AI4Science;AI4Chemistry;Tool-Augmented LLMs
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
Comprehensive evaluation of LLM agent on chemistry tasks and reveal findings and challenges.
@misc{
yu2024tooling,
title={Tooling or Not Tooling? The Impact of Tools on Language Agents for Chemistry Problem Solving},
author={Botao Yu and Frazier N. Baker and Ziru Chen and Garrett Herb and Boyu Gou and Daniel Adu-Ampratwum and Xia Ning and Huan Sun},
year={2024},
url={https://openreview.net/forum?id=Q6S9gZvY6j}
}
Large language models (LLMs) have shown promise in various domains but face challenges in chemistry due to limited domain knowledge and computational capabilities. To address these issues, tool-augmented language agents like ChemCrow and Coscientist have been developed. However, their evaluations remain narrow in scope, leaving an unclear understanding of how these tool-augmented agents perform across various real-world applications. In this study, we conduct a comprehensive evaluation to bridge this gap. Specifically, we develop ChemAgent, the most capable chemistry agent to date, equipped with 29 tools capable of handling a wide spectrum of tasks. We then conduct a comprehensive assessment across three datasets, namely SMolInstruct, MMLU-chemistry, and GPQA-chemistry, which can be categorized into specialized chemistry tasks and general chemistry questions. Surprisingly, tool-augmented agents do not consistently outperform the base LLM without tools, and the impact of tool augmentation is highly task-dependent: It provides substantial gains in specialized chemistry tasks but potentially hinders performance in general chemistry questions. We further engage domain experts and conduct error analysis, revealing that errors in general chemistry questions primarily occur due to minor inaccuracies at intermediate stages of the problem-solving process, highlighting the need for further research into balancing tool use with intrinsic reasoning abilities, to maximize the effectiveness of language agents in chemistry.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Botao_Yu1 ~Frazier_N._Baker1 ~Ziru_Chen1 ~Garrett_Herb1 ~Boyu_Gou1 ~Daniel_Adu-Ampratwum1 ~Xia_Ning1 ~Huan_Sun1
Botao Yu Frazier N. Baker Ziru Chen Garrett Herb Boyu Gou Daniel Adu-Ampratwum Xia Ning Huan Sun
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Chemistry LLMs Agent AI4Science AI4Chemistry Tool-Augmented LLMs
I certify that there is no acknowledgement section in this submission for double blind review.
yu|tooling_or_not_tooling_the_impact_of_tools_on_language_agents_for_chemistry_problem_solving
/pdf/fa2c36e16efc73f8b317ea6de527c0fc51df0392.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Tooling or Not Tooling? The Impact of Tools on Language Agents for Chemistry Problem Solving
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission