Robustness of hyperparameter tuning
One option is not to measure the performance of the hyperparameters on the loss function of the training data but measure performance of the hyperparameters on the elevation metric on the validation data. The end goal of the most machine learning systems is the ability to predict on unseen data. Focusing on &quot;best solution&quot; as measured by loss function on training will lead to overfitting / non-robust solutions.