LLMs are opening new possibilities for leveraging natural language processing in chemistry and other scientific fields. These models can access and generate chemical information, potentially assisting researchers with tasks such as predicting molecular properties, extracting structured data from text, and even designing new molecules. However, using LLMs in chemical research comes with unique challenges. One prominent issue is "hallucination," where the model produces outputs that are confidently incorrect, often due to gaps or inconsistencies in its training data . Hallucinations present a substantial obstacle in chemistry, where even minor inaccuracies can lead to significant misinterpretations in predicting molecular properties or reactions . To fully integrate LLMs into chemical research workflows, these hallucinations must be addressed and it is critical to improve the models' ability to better handle chemical data.