Methods for optimizing hyperparameters Grid search is an effective strategy when the response is sensitive to all the hyperparameters. However, it has poor scaling with the number of hyperparameters (N d , where N is the number of grid points per hyperparameter and d is the number of hyperparameters), so when only a small subset of hyperparameters are relevant, random search is more efficient. Model-based search algorithms construct surrogate models of the mapping between the hyperparameters and the model response which are cheap to evaluate and optimize, and use these models to guide hyperparameters to test. Examples include Bayesian optimization with either a Gaussian process or a tree Parzen estimator (TPE) as the surrogate model. The third class of optimization algorithms is population algorithms, which include evolutionary algorithms, 32,34 particle swarm optimization 33,34 and covariance matrix adaption, 35 these will not be explored here further.