Robustness of hyperparameter tuning I use a Bayesian hyperparameter (HP) optimization approach (BOHB) to tune a deep learning model. However, the resulting model is not robust when repeatedly applied to the same data. I know, I could use a seed to fix the parameter initialization, but I wonder if there are HP optimization approaches that already account for robustness.
To illustrate the problem, let's consider a one-layer neural network with only one HP: the hidden size (h). The model performs well with a small h. With a larger h, the results start to fluctuate more, maybe due to a more complex loss landscape; the random initialization of the parameters can lead to a good performance, or to a very bad performance if the optimizer gets stuck in a local minimum (which happens more often due to the complex loss landscape). The loss vs h plot could look something like this:

I would prefer the 'robust solution', while the 'best solution' is selected by the HP optimizer algorithm. Are there HP optimization algorithms that account for the robustness? Or how would you deal with this problem?
One option is not to measure the performance of the hyperparameters on the loss function of the training data but measure performance of the hyperparameters on the elevation metric on the validation data. The end goal of the most machine learning systems is the ability to predict on unseen data. Focusing on &quot;best solution&quot; as measured by loss function on training will lead to overfitting / non-robust solutions.