optimal combination of hyper parameters and model selection This is a general question which often comes up when tuning deep learning and machine learning algorithms such as recurrent neural network, multilayer perceptron or SVM etc. 

When we tune the hyper parameters of a deep learning model every possible combination of hyper parameters results in a different model. And we select an optimal combination based on the loss curves. What exactly is an optimal combination of hyper parameters?

My question is exactly this: There is an infinite number of combinations of hyperparams possible. We know that there are many possible configurations of hyperparams possible that give similar generalization error. What should the model selection decision be based upon?
And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?
"What exactly is an optimal combination of hyper parameters?" The combination of hyperparameters that produces the lowest possible error on unseen data for that model architecture.

"What should the model selection decision be based upon?" The best way to estimate error on unseen data that I know of is k-fold cross-validation. I examine the mean and standard deviation of the error across the k-folds; in most cases, I select the model with the smallest standard deviation among the models with the best mean error. 

"And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?" As far as I know, one can never know what combination of hyperparameters will produce the lowest possible error on unseen data. In my experience, a good search strategy will get you close enough to the optimal combination so that further search is not worth the effort.