This work explores whether large language models (LLM) truly understand the intrinsic chemistry of molecules. To investigate this, we evaluated the consistency of LLMs across chemistry tasks using different molecular representations, such as SMILES strings and IU-PAC names. Our findings reveal that LLMs exhibit low consistency between these representations, even when trained on carefully controlled one-to-one mapped data.