Title: Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuning,
Abstract: Machine learning applications often require hyperparameter tuning. The
hyperparameters usually drive both the efficiency of the model training process
and the resulting model quality. For hyperparameter tuning, machine learning
algorithms are complex black-boxes. This creates a class of challenging
optimization problems, whose objective functions tend to be nonsmooth,
discontinuous, unpredictably varying in computational expense, and include
continuous, categorical, and/or integer variables. Further, function
evaluations can fail for a variety of reasons including numerical difficulties
or hardware failures. Additionally, not all hyperparameter value combinations
are compatible, which creates so called hidden constraints. Robust and
efficient optimization algorithms are needed for hyperparameter tuning. In this
paper we present an automated parallel derivative-free optimization framework
called \textbf{Autotune}, which combines a number of specialized sampling and
search methods that are very effective in tuning machine learning models
despite these challenges. Autotune provides significantly improved models over
using default hyperparameter settings with minimal user interaction on
real-world applications. Given the inherent expense of training numerous
candidate models, we demonstrate the effectiveness of Autotune's search methods
and the efficient distributed and parallel paradigms for training and tuning
models, and also discuss the resource trade-offs associated with the ability to
both distribute the training process and parallelize the tuning process.