Notably, on the "perovskites" task, which incorporates structural details, language models such as AlchemBERT outperform CrabNet 11 significantly. This is likely because CrabNet 11 relies solely on compositional information. However, incorporating even basic structural in-formation can lead to significant improvements, as all the data pertain to perovskite-like structures. On certain specific tasks (See the Matbench 2 leaderboard for details), Alchem-BERT even surpass traditional structure-based models like CGCNN, 2,12 DeeperGATGNN, MegNet, DimeNet++, and knowledge-driven models such as MODNet. These results suggest that while AlchemBERT may not fully capture the structural in-