Tunability: Importance of Hyperparameters of Machine Learning Algorithms
Philipp Probst Bernd Bischl Anne-Laure Boulesteix






Tunability: Importance of Hyperparameters of Machine Learning Algorithms
    by Philipp Probst, Anne-Laure Boulesteix and Bernd Bischl
    December 30, 2023
========================================================================


Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. 
Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform and six common machine learning algorithms. We apply our measures to assess the tunability of their parameters. Our results yield default values for hyperparameters and enable users to decide whether it is worth conducting a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning. 




§ INTRODUCTION


Machine learning (ML) algorithms such as gradient boosting, random forest and neural networks for regression and classification involve a number of hyperparameters that have to be set before running them. In contrast to direct, first-level model parameters, which are determined during training, these second-level tuning parameters often have to be carefully optimized to achieve maximal performance. A related problem exists in many other algorithmic areas, e.g., control parameters in evolutionary algorithms <cit.>. 

In order to select an appropriate hyperparameter configuration for a specific dataset at hand, users of ML algorithms can resort to default values of hyperparameters that are specified in implementing software packages or manually configure them, for example, based on recommendations from the literature, experience or trial-and-error. 

Alternatively, one can use hyperparameter tuning strategies, which are data-dependent, second-level optimization procedures <cit.>, which try to minimize the expected generalization error of the inducing algorithm over a hyperparameter search space of considered candidate configurations, usually by evaluating predictions on an independent test set, or by running a resampling scheme such as cross-validation <cit.>. For a recent overview of tuning strategies, see, e.g., <cit.>. 

These search strategies range from simple grid or random search <cit.> to more complex, iterative procedures such as
Bayesian optimization <cit.> or iterated F-racing <cit.>. 

In addition to selecting an efficient tuning strategy, 
the set of tunable hyperparameters and their corresponding ranges, scales and potential prior distributions for subsequent sampling have to be determined by the user. Some hyperparameters might be safely set to default values, if they work well across many different scenarios. 
Wrong decisions in these areas can inhibit either the quality of the resulting model or at the very least the efficiency and fast convergence of the tuning procedure.
This creates a burden for: 


  * ML users – Which hyperparameters should be tuned and in which ranges?

  * Designers of ML algorithms – How do I define robust defaults?

We argue that many users, especially if they do not have years of practical experience in the field, here often rely on heuristics or spurious knowledge. It should also be noted that designers of fully automated tuning frameworks face at least very similar problems. 
It is not clear how these questions should be addressed in a data-dependent, automated, optimal and objective manner. In other words, the scientific community not only misses answers to these questions for many algorithms but also a systematic framework, methods and criteria, which are required to answer these questions.

With the present paper we aim at filling this gap and formalize the problem of parameter tuning from a statistical point of view, in order to simplify the tuning process for less experienced users and to optimize decision making for more advanced processes.  

After presenting related literature in section <ref>, 
we define theoretical measures for assessing the impact of tuning in section <ref>. 
For this purpose we (i) define the concept of default hyperparameters, (ii) suggest measures for quantifiying the tunability 
of the whole algorithm and specific hyperparameters based on the differences between the performance of default hyperparameters and the performance 
of the hyperparameters when this hyperparameter is set to an optimal value. Then we (iii) address the tunability of hyperparameter combinations and joint gains, (iv) provide theoretical definitions for an appropriate hyperparameter space on which tuning 
should be executed and (v) propose procedures to estimate these quantities based on the results of a benchmark study  with 
random hyperparameter configurations with the help of surrogate models. In sections <ref> and <ref> we illustrate these 
concepts and methods through an application. For this purpose we use benchmark results of six machine learning algorithms 
with different hyperparameters which were evaluated on 38 datasets from the OpenML platform. Finally, in the last section <ref> 
we conclude and discuss the results. 



§ RELATED LITERATURE



To the best of our knowledge, only a limited amount of articles address the problem of tunability and generation of tuning search spaces. 
<cit.> compute the relevance of the hyperparameters of neural networks and conclude that some 
are important on all datasets, while others are only important on some datasets. Their conclusion is primarily visual and used as an argument for why random search works better than grid search when tuning neural networks.

A specific study for decision trees was conducted by <cit.> who apply standard tuning techniques to decision trees on 102 datasets and calculate differences of accuracy between the tuned algorithm and the algorithm with default hyperparameter settings. 

A different approach is proposed by <cit.>, which aims at identifying the most important hyperparameters via forward selection. 
In the same vein, <cit.> present an ablation analysis technique, which aims at identifying the hyperparameters that contribute the most to improved performance after tuning. 
For each of the considered hyperparameters, they compute the performance gain that can be achieved by changing its value from the initial value to the value specified in the target configuration which was determined by the tuning strategy. This procedure is iterated in a greedy forward search. 

A more general framework for measuring the importance of single hyperparameters is presented by <cit.>. 
After having used a tuning strategy such as sequential model-based optimization, a functional ANOVA approach is used for measuring the importance of hyperparameters. 

These works concentrate on the importance of hyperparameters on single datasets, mainly to retrospectively explain what happened during an already concluded tuning process. 
Our main focus is the generalization across multiple datasets in order to facilitate better general understanding of hyperparameter effects and better decision making for future experiments. 
In a recent paper <cit.> pose very similar questions to ours to assess the importance of hyperparameters across datasets. We compare  it to our approach  in section <ref>.

Our framework is based on using surrogate models, also sometimes called empirical performance models, which allow estimating the performance of arbitrary hyperparameter configurations based on a limited number of prior experiments. 
The idea of surrogate models is far from new, as it constitutes the central idea of Bayesian optimization for hyperparameter search but is also used, for example, in <cit.> for increasing the speed of an ablation analysis and by <cit.> for speeding up the benchmarking of tuning strategies. 



§ METHODS FOR ESTIMATION OF DEFAULTS, TUNABILITY AND RANGES





 §.§ General notation


Consider a target variable Y, a feature vector X, and an unknown joint distribution P on (X,Y), from which we have sampled a dataset 𝒯 of n observations. 
A machine learning (ML) algorithm now learns the functional relationship between X and Y by producing a prediction model f̂(X, θ), controlled by the k-dimensional hyperparameter configuration θ = (θ_1, ...,θ_k) from the hyperparameter search space Θ = Θ_1 × ... ×Θ_k. 
In order to measure prediction performance pointwise between the true label Y and its prediction f̂(X, θ), we define a loss function L(Y, f̂(X, θ)). 
We are naturally interested in estimating the expected risk of the inducing algorithm, w.r.t. θ on new data, also sampled from 𝒫:
R(θ) = E(L(Y,f̂(X, θ))| 𝒫). 
This mapping encodes, given a certain data distribution, a certain learning algorithm and a certain performance measure, the numerical quality for any hyperparameter configuration θ. Given m different datasets (or data distributions) 𝒫_1,...,𝒫_m, we arrive at m hyperparameter risk mappings

    R^(j)(θ) := E(L(Y,f̂(X, θ))| 𝒫_j),      j = 1,...,m.

For now, we assume all R^(j)(θ) to be known, and show how to estimate them in section <ref>. 



 §.§ Optimal configuration per dataset and optimal defaults



We first define the best hyperparameter configuration for dataset j as


    θ^(j)⋆ := θ∈Θ R^(j)(θ).


Defaults settings are supposed to work well across many different datasets and are usually provided by software packages, in an often ad hoc or heuristic manner. 
We propose to define an optimal default configuration, based on an extensive number of empirical experiments on m different benchmark datasets, by 

    θ^⋆ := θ∈Θ g(R^(1)(θ), ..., R^(m)(θ)).


Here, g is a summary function that has to be specified. Selecting the mean (or median as a more robust candidate) would imply 
minimizing the average  (or median)  risk over all datasets. 

The measures R^(j)(θ) could potentially be scaled appropriately beforehand in order to make them more commensurable between datasets, e.g., one could scale all R^(j)(θ) to [0,1] by substracting the result of a very simple baseline like a featureless dummy predictor and dividing this difference by the absolute difference between the risk of the best possible result (as an approximation of the Bayes error) and the result of the very simple baseline predictor. Or one could produce a statistical z-score by subtracting the mean and dividing by the standard deviation from all experimental results on the same dataset <cit.>.

The appropriateness of the scaling highly depends on the performance measure that is used. 
One could, for example, argue that the AUC does not have to be scaled as an improvement from 0.5 to 0.6 can possibly be seen as important as an improvement from 0.8 to 0.9. 
On the other hand, averaging the mean squared error on several datasets does not make a lot of sense, as the scale of the outcome of different regression problems can be very different. 
Then scaling or using another measure such as R^2 seems essential. 



 §.§ Measuring overall tunability of a ML algorithm



A general measure of the tunability of an algorithm per dataset can then be computed based on the difference between the risk of an overall reference configuration (e.g., either the software defaults or definition (<ref>)) and the risk of the best possible configuration on that dataset:


    d^(j) := R^(j)(θ^⋆) - R^(j)(θ^(j)⋆) ,  j = 1,...,m.


For each algorithm, this gives rise to an empirical distribution of performance differences over datasets, which might be directly visualized or summarized to an aggregated tunability measure d by using mean, median or quantiles. 



 §.§ Measuring tunability of a specific hyperparameter
 

The best hyperparameter value for one parameter i on dataset j, when all other parameters are set to defaults from θ^⋆ := (θ_1^⋆, ...,θ_k^⋆), is denoted by


    θ^(j)⋆_i := θ∈Θ, θ_l = θ_l^⋆∀ l ≠ i R^(j)(θ).


A natural measure for tunability of the i-th parameter on dataset j is then the difference in risk between the above and our default reference configuration:

    d_i^(j) := R^(j)(θ^⋆) - R^(j)(θ^(j)⋆_i),  j = 1,...,m, i = 1,...,k.


Furthermore, we define d_i^(j),rel = d_i^(j)/d^(j) as the fraction of performance gain, when we only tune i compared to tuning the complete algorithm, on dataset j.
Again, one can calculate the mean, the median or quantiles of these two differences over the n datasets, to get a notion of the overall tunability d_i of this parameter. 



 §.§ Tunability of hyperparamater combinations and joint gains
 

Let us now consider two hyperparameters indexed as i_1 and i_2. To measure the tunability with respect to these two parameters, we define

    θ^(j)⋆_i_1,i_2 := θ∈Θ, θ_l = θ_l^⋆∀ l ∉{i_1, i_2} R^(j)(θ),

i.e., the θ-vector containing the default values for all hyperparameters other than  i_1 and i_2, and the optimal combination of values for the i_1-th and i_2-th components of θ.

Analogously to the previous section, we can now define the tunability of the set (i_1, i_2) as the gain over the reference default on dataset j as:

    d^(j)_i_1, i_2 :=      R^(j)(θ^*) - R^(j)(θ^(j)⋆_i_1,i_2).


The joint gain which can be expected when tuning not only one of the two hyperparameters individually, but both of them jointly, on a dataset j, can be expressed by:

    g^(j)_i_1, i_2 :=    min{(R^(j)(θ^(j)⋆_i_1)), (R^(j)(θ^(j)⋆_i_2))} - R^(j)(θ^(j)⋆_i_1,i_2).


Furthermore, one could be interested in whether this joint gain could simply be reached by tuning both parameters i_1 and i_2 in a univariate fashion sequentially, either in the order i_1 → i_2 or i_2 → i_1, and what order would be preferable.
For this purpose one could compare the risk of the hyperparameter value that results when tuning them together R^(j)(θ^(j)⋆_i_1,i_2) with the risks of the hyperparameter values that are obtained when tuning them sequentially, that means R^(j)(θ^(j)⋆_i_1 → i_2) or R^(j)(θ^(j)⋆_i_2 → i_1), which is done for example in <cit.>.

Again, all these measures should be summarized across datasets, resulting in d_i_1, i_2 and g_i_1, i_2. Of course, these approaches can be further generalized by considering combinations of more than two parameters. 



 §.§ Optimal hyperparameter ranges for tuning


A reasonable hyperparameter space Θ^⋆ for tuning should include the optimal configuration θ^(j)⋆ for dataset j with high probability. 
We denote the p-quantile of the distribution of one parameter regarding the best hyperparameters on each dataset (θ^(1)⋆)_i, ..., (θ^(m)⋆)_i as q_i,p. 
The hyperparameter tuning space can then be defined as:


    Θ^⋆ := {θ∈Θ | ∀ i ∈{1,...,k}: θ_i ≥ q_i, p_1θ_i ≤ q_i, p_2},

with p_1 and p_2 being quantiles which can be set for example to the 5 % quantile and the 95 % quantile. 
This avoids focusing too much on outlier datasets and 
makes the definition of the space independent from the number of datasets.

The definition above is only valid for numerical hyperparameters. In case of categorical variables one could use similar rules, for example 
only including hyperparameter values that were at least once or in at least 10 % of the 
datasets the best possible hyperparameter setting. 



 §.§ Practical estimation



In order to practically apply the previously defined concepts, two remaining issues need to be addressed: a) We need to discuss how to obtain R^(j)(θ); and b) in (<ref>) and (<ref>) a multivariate optimization problem (the minimization) needs to be solved[All other previous optimization problems are univariate or two-dimensional and can simply be addressed by a simple technique like a fine grid search ]. 

For a) we estimate R^(j)(θ) by using surrogate models R̂^(j)(θ), and replace the original quantity by its estimator in all previous formulas. Surrogate models for each dataset j are based on a meta dataset. This is created by evaluating a large number of configurations of the respective ML method. The surrogate regression model then learns to map a hyperparameter configuration to estimated performance. For b) we  solve the optimization problem – now cheap to evaluate, because of the surrogate models – through black-box optimization.



§ EXPERIMENTAL SETUP



In this section we give an overview about the experimental setup that is used for obtaining  surrogate models, tunability measures and tuning spaces.



 §.§ Datasets from the OpenML platform


Recently, the OpenML project <cit.> has been created as a flexible online platform that allows ML scientists to share their data, corresponding tasks and results of different ML algorithms.  
We use a specific subset of carefully curated classification datasets from the OpenML 
platform called OpenML100 <cit.>.
For our study we only use the 38 binary classification tasks that do not contain any missing values. 



 §.§ ML Algorithms


The algorithms considered in this paper are common methods for supervised learning. 
We examine elastic net (), decision tree (), k-nearest neighbors (), 
support vector machine (), random forest () and gradient boosting (). For more details about the used software packages see <cit.>. 
An overview of their considered hyperparameters is displayed in Table <ref>, including respective data types, box-constraints and a potential transformation function.



In the case of , the underlying package only supports numerical features, so we opted for a dummy feature encoding for categorical features, which is performed internally by the underlying packages for  and .

Some hyperparameters of the algorithms are dependent on others. We take into account these dependencies and, for example, only 
sample a value for  for the support vector machine if the radial kernel was sampled beforehand.



 §.§ Performance estimation



Several measures are regarded throughout this paper, either for evaluating our considered classification models that should be tuned, or for evaluating our surrogate regression models. As no optimal measure exists, we will compare several of them.
In the classification case, we consider AUC, accuracy and brier score.
In the case of surrogate regression, we consider R^2, which is directly proportional to the regular mean squared error but scaled to [0,1] and explains the gain over a constant model estimating the overall mean of all data points. 
We also compute Kendall's tau as a ranking based measure for regression. 

The performance estimation for the different hyperparameter experiments is computed through 10-fold cross-validation. For the comparison of surrogate models 10 times repeated 10-fold cross-validation is used.



 §.§ Random Bot sampling strategy for meta data



To reliably estimate our surrogate models we need enough evaluated configurations per classifier and data set.
We sample these points from independent uniform distributions where the respective support for each parameter is displayed in Table <ref>. Here, uniform refers to the untransformed scale, so we sample uniformly from the interval [Lower, Upper] of Table <ref>.

In order to properly facilitate the automatic computation of a large database of hyperparameter experiments, we implemented a so called 
OpenML bot. 
In an embarrassingly parallel manner it chooses in each iteration a 
random dataset, a random classification algorithm, samples a random configuration and evaluates it via cross-validation. 
A subset of 500000 experiments for each algorithm and all datasets are used for our analysis here.[30 for each dataset for ]
More technical details regarding the random bot, its setup and results  can be obtained in <cit.>, furthermore, for simple and permanent access the results of the bot are stored in a figshare repository <cit.>. 



 §.§ Optimizing surrogates to obtain optimal defaults

Random search is also used for our black-box optimization problems in section <ref>. For the estimation of the defaults for each algorithm 
we randomly sample 100000 points in the hyperparameter space as defined in Table <ref> and determine the configuration with the minimal average risk. 
The same strategy with 100000 random points is used to obtain the best hyperparameter setting on each dataset that is needed for the estimation of the tunability of an algorithm. 
For the estimation of the tunability of single hyperparameters we also use 100000 random points for each parameter, while for the tunability of combination of hyperparameters we only use 10000 random points to reduce runtime as this should be enough to cover 2-dimensional hyperparameter spaces. 

Of course one has to be careful with overfitting here, as our new defaults are chosen with the help of the same datasets that are used to determine the performance. 
Therefore, we also evaluate our approach via a “10-fold cross-validation across datasets”. 
Here, we repeatedly calculate the optimal defaults based on 90% “training datasets” and evaluate the package defaults and our optimal defaults – the latter induced from the training data sets – on the surrogate models of the remaining 10% “test datasets”, and compare their difference in performance.



 §.§ The problem of hyperparameter dependency


Some parameters are dependent on other superordinate hyperparameters and are only relevant if the parameter value of this superordinate parameter was set to a specific value. 
For example  in  only makes sense if the  was set to “radial“ or  only makes sense if the kernel was set to “polynomial“. 
Some of these subordinate parameters might be invalid/inactive in the reference default  configuration, rendering it impossible to univariately tune them in order to compute their tunability score.
In such a case we set the superordinate parameter to a value which makes the subordinate parameter active, compute the optimal defaults for the rest of the parameters and compute the tunability score for the subordinate parameter with these defaults.



 §.§ Software details


All our experiments are executed in R and are run through a combination of custom code from our random bot <cit.>, the  R package <cit.>,  <cit.> and  <cit.> for parallelization. All results are uploaded to the OpenML platform and there publicly available for further analysis. 
 is also used to compare and fit all surrogate regression models. 
The fully reproducible R code for all computations and analyses of our paper can be found on the github page: <https://github.com/PhilippPro/tunability>.
We also provide an interactive shiny app under <https://philipppro.shinyapps.io/tunability/>, which displays all results of the following section in a potentially more convenient, interactive fashion and which can simply be accessed through a web browser.





§ RESULTS AND DISCUSSION



We calculate all results for AUC, accuracy and brier score but mainly discuss AUC results here. Tables and  figures  for the other measures can be accessed in the  Appendix  and in our interactive shiny application.



 §.§ Surrogate models



We compare different possible regression models as candidates for our surrogate models: 
the linear model (), a simple decision tree (),  () and random forest ()[We also tried  <cit.>, which provided good results but the algorithm had some technical problems for some combinations of datasets and algorithms. We did not include gaussian process which is one of the standard algorithms for surrogate models as it cannot handle categorical variables.] 
All algorithms are run with their default settings. 
We calculate 10 times repeated 10-fold cross-validated regression performance measures R^2 and Kendall's tau per dataset, and average these across all datasets[In case of  four datasets did not provide results for one of the surrogate models and were not used.].
Results for AUC are displayed in Figure <ref>. 

A good overall performance is achieved by  with qualitatively similar results for other classification performance measures (see  Appendix). 
In the following we use random forest as surrogate model because it performs reasonably well and is already an established algorithm for surrogate models in the literature <cit.>.



  §.§.§ Optimal defaults and tunability


Table <ref> displays our mean tunability results for the algorithms as defined in formula (<ref>) w.r.t. package defaults ( column) and our optimal defaults ().
It also displays the improvement per algorithm when moving from package defaults to optimal defaults (), which was positive overall. 
This also holds for  and  although the package defaults are data dependent, which we currently cannot model 
( = 1/p for  and  = √(p) for ). 
From now on, when discussing tunability, we will only do this w.r.t. our optimal defaults.

Clearly, some algorithms such as  and  are much more tunable than the others, while  is the algorithm with the smallest tunability, which is in line with common knowledge in the web community. 
In Figure <ref> modified boxplots of the tunabilities are depicted. For each ML algorithm, some outliers are visible, which indicates that tuning has a much higher impact on some specific datasets. 







  §.§.§ Tunability of specific hyperparameters


In Table <ref> the mean tunability (regarding the AUC) of single hyperparameters as defined in Equation (<ref>) 
in section <ref> can 
be seen. 
From here on, we will refer to tunability only with respect to optimal defaults.



For   seems to be more tunable than . 
In  the  and  parameters seem to be 
the most important ones for tuning. 
 in the  algorithm is very tunable w.r.t. package defaults, but not regarding optimal defaults. 
In  the biggest gain in performance can 
be achieved by tuning the ,  or , while the  parameter does not seem 
to be very tunable. In   is the most tunable parameter. 
For  there are two parameters that are quite tunable:  and the .  specifies if 
a tree or a linear model is trained.
The cross-validated results can be seen in Table <ref> in the  Appendix, they are quite similar to the 
non cross-validated results and for all parameters slightly higher. 

Instead of looking only at the average, as in Table <ref>, one could also be interested in the distribution of the tunability of each dataset. 
As an example, Figure <ref> shows the tunability of each parameter of  in a boxplot. This gives a more in-depth insight 
into the tunability, makes it possible to detect outliers and to examine the skewness. 





  §.§.§ Hyperparameter space for tuning


The hyperparameter space for tuning, as defined in Equation (<ref>) in section <ref> and based on the 0.05 and 0.95 quantiles, is displayed in Table <ref>.
All optimal defaults are contained in this hyperparameter space while some of the package defaults are not. 

As an example, Figure <ref> displays the full histogram of the best values of  of the random forest over all datasets. 
Note that for quite a few data sets much higher values than the package defaults seem advantageous. Analogous histograms for other parameters are available through the shiny app.





  §.§.§ Tunability of hyperparameter combinations


As an example, Table <ref> displays the average tunability d_i_1,i_2 of all 2-way hyperparameter combinations for .
Obviously, the increased flexibility in tuning a 2-way combination enables larger improvements when compared with the tunability of one of the respective individual parameters. 

In Table <ref> the joint gain of tuning two hyperparameters g_i_1,i_2 instead of only the best as defined in section <ref> can be seen. 
The parameters  and  have the biggest joint effect, which is not very surprising, as they are closely related: 
 is the minimum number of observations that must exist in a node in order for a split to be attempted and 
 the minimum number of observations in any terminal leaf node. 
If a higher value of  than 
the default performs better on a dataset it is possibly not enough to set it higher without also increasing , so the strong relationship is quite clear. 
Again, further figures for other algorithms are available through the shiny app. 







§ CONCLUSION AND DISCUSSION



Our paper provides concise and intuitive definitions for optimal defaults of ML algorithms and the impact of tuning them either jointly, tuning individual parameters or combinations, all based on the general concept of surrogate empirical performance models.
Tunability values as defined in our framework are easily and directly interpretable as how much performance can be gained by tuning this hyperparameter?. This allows direct comparability of the tunability values across different algorithms.

In an extensive OpenML benchmark, we computed optimal defaults for 
elastic net, decision tree, k-nearest neighbors, SVM, random forest and xgboost and quantified their tunability and the tunability of their individual parameters. 
This – to the best of our knowledge – has  never  been provided before in such a principled manner.
Our results are often in line with common knowledge from literature and our method itself now allows an analogous analysis for other or more complex methods. 

Our framework is based on the concept of default hyperparameter values, which can be seen both as an advantage (default values are a valuable output of the approach) and as an inconvenience (the determination of the default values is an additional analysis step and needed as a reference point for most of our measures).

We now compare our method with <cit.>. 
In contrast to us, they apply the functional ANOVA framework from <cit.> on a surrogate random forest to assess the importance of hyperparameters regarding empirical performance of a support vector machine, random forest and adaboost, which results in numerical importance scores for individual hyperparameters.
Their numerical scores are - in our opinion - less directly interpretable, but they do not rely on defaults as a reference point, which one might see as an advantage.
They also propose a method for calculating hyperparameter priors, combine it with the tuning procedure hyperband, and assess the performance of this new tuning procedure.
In contrast, we define and calculate ranges for all hyperparameters.
Setting ranges for the tuning space can be seen as a special case of a prior distribution - the uniform distribution on the specified hyperparameter space. 
Regarding the experimental setup, we compute more hyperparameter runs (around 2.5 million vs. 250000), but consider only the 38 binary classification datasets of OpenML100 while <cit.> use all the 100 datasets which also contain multiclass datasets. 
We evaluate the performance of different surrogate models by 10 times repeated 10-fold cross-validation to choose an appropriate model and to assure that it performs reasonably well. 

Our study has some limitations that could be addressed in the future:
a) We only considered binary classification, where we tried to include a wider variety of datasets from different domains. 
In principle this is not a restriction as our methods can easily be applied to multiclass classification, regression, survival analysis or even algorithms not from machine learning whose empirical performance is reliably measurable on a problem instance. 
b) Uniform random sampling of hyperparameters might not scale enough for very high dimensional spaces, and a smarter sequential technique might be in order here, see <cit.> for an potential approach of sampling across problem instances to learn optimal mappings from problem characteristics to algorithm configurations. 
c) We currently are learning static defaults, which cannot depend on dataset characteristics (like number of features, or further statistical measures) as in meta-learning. Doing so might improve performance results of optimal defaults considerably, but would require a more complicated approach. 
d) Our approach still needs initial ranges to be set, in order to run our sampling procedure. Only based on these wider ranges we can then compute more precise, closer ranges. 



§ ACKNOWLEDGEMENTS

We would like to thank Joaquin Vanschoren for support regarding the OpenML platform and Andreas Müller, Jan van Rijn, Janek Thomas and Florian Pfisterer for reviewing and useful comments. Thanks to Jenny Lee for language editing. 








§ APPENDIX A. RESULTS FOR ACCURACY AND BRIER SCORE




























