Until recently, BERT and its variants have demonstrated exceptional performance on language benchmarks such as GLUE (General Language Understanding Evaluation) and SQuAD (Stanford Question Answering Dataset). However, their application to Matbench has been largely overlooked. In this work, we address this gap by providing a baseline for lightweight language models and subsequently evaluating whether billion-parameter language models offer significant advantages. Furthermore, our study is particularly focused on exploring whether language models can effectively handle not only compositional tasks but also structural tasks. Our findings reveal that language models surpass many humandesigned models. As a result, this research suggests a future where physicists might simply describe a structure in natural language, eliminating the need for designing intricate models or laboriously screening material features.