Hyperparameter optimization, ensembling instead of selecting with CV criteria While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the `scikit-learn' API, for which the end result is a single point on the hyperparameter space, for which the performance is the best according to chosen metric. It looks like a lot of information is trashed, we are probing the performance landscape of the hyperparameters and keeping a single point estimate. Information such as having many modes, local maximas, maybe far apart. An ensembler method could make good use of such information. Is there a theory built over ensembling a diversity of hyperparameter estimates?
I think you are looking for hyperopts, Optuna and Gpopy for Hyperparameters search without burning much of CPUs.