optimal combination of hyper parameters and model selection
"What exactly is an optimal combination of hyper parameters?" The combination of hyperparameters that produces the lowest possible error on unseen data for that model architecture.

"What should the model selection decision be based upon?" The best way to estimate error on unseen data that I know of is k-fold cross-validation. I examine the mean and standard deviation of the error across the k-folds; in most cases, I select the model with the smallest standard deviation among the models with the best mean error. 

"And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?" As far as I know, one can never know what combination of hyperparameters will produce the lowest possible error on unseen data. In my experience, a good search strategy will get you close enough to the optimal combination so that further search is not worth the effort.