To explore this, we evaluate the consistency of LLMs in chemistry tasks across different string-based molecular representations, focusing on two widely used formats: SMILES strings and IUPAC names 14 . Our findings reveal that LLMs exhibit strikingly low consistency between these representations, even when trained on carefully curated, one-to-one mapped datasets. To address this limitation, we introduce a sequence-level Kullback-Leibler (KL) divergence 15 loss during training, aimed at encouraging LLMs to produce more consistent outputs across representations. While the KL divergence loss improves consistency, it also amplifies consistently incorrect predictions. Further analysis demonstrates the orthogonality between consistency and accuracy. These results suggest that LLMs still struggle with fully capturing the fundamental chemistry underlying molecular representations.