New machine learning (ML) models for the material science community are required to "beat" increasingly formidable accuracy baselines. While exceeding the accuracy baselines might be considered natural within the computer science community, they have visible influence on the publication of new ML models in materials research publications. For example, the leaderboard of the benchmarking website Matbench showcases ML models with prediction accuracies as low as 17 meV mean absolute error (MAE) for predicting the formation energy data (as of 3 rd March 2025) in MP. Similar results are displayed in the JARVIS leaderboard. While such exceedingly low MAE value can be motivating for future innovations in ML, it must also motivate us to question the data itself.