How to be sure of a good solution in hyperparameter optimization I have a very expensive black-box function and at least 15 parameters that I want to explore (usually 5-6 at a time). So far I have tried Genetic Algorithms and Gaussian Process Surrogate Optimization (bayesian optimization). Both methods work fine and efficiently up to 5-6 parameters. However, I am having an issue on how can one be sure that a solution found by GA/GPSO is a good solution, without basically knowing nothing about the output space. I don't really want to go for an exhaustive grid search since even with 5 parameters, it requires incredible amounts of evaluations, yet I cannot come up with anything better. 

I would also very much appreciate suggestions/keywords for literature reading, or event different methods for hyperparameter optimizations.

Thanks in advance
The short answer: You can't.  These are heuristics.  There are no guarantees.  They might fail badly; and there's no reliable way to detect that.