\section{Introduction}

Data scientists have a wealth of algorithms at their disposal when designing machine learning pipelines. These include methods for prepossessing, data transformation, feature selection, and, finally, the choice of machine learning algorithms. Each step may also include numerous parameter settings that must be explored and optimized for each dataset. As more tools get developed, the space of possible combinations of algorithms grows. As datasets get larger, the time it takes to explore the optimal combination of these steps also grows. In response, Automated Machine Learning (AutoML) tools have been increasingly viable as new sophisticated search algorithms are developed and computers grow in computational power. Over the years, multiple AutoML algorithms were developed, including TPOT \cite{OlsonGECCO2016}, Auto-Sklearn 1 \cite{feurer-neurips15a}, and Auto-Sklearn 2 \cite{feurer-arxiv20a}, H2O AutoML \cite{H2OAutoML20}. 



AutoML aims to create an accessible yet powerful off-the-shelf algorithm to serve as a jump start for a wide array of problems. AutoML aims to produce higher-performing models quicker by more efficiently searching through the space of possible models. These features can increase the accessibility of machine learning analysis to a broader range of people.

When evaluating AutoML tools, it is important to benchmark them against diverse datasets. There have been several benchmarks that compile challenging datasets. Examples include Open Machine Learning \cite{OpenML2013}, University of California Irvine (UCI) machine learning repository \cite{Dua:2019}, Library for Support Vector Machines (LIBSVM) \cite{chang2011libsvm},Penn Machine Learning Benchmark (PMLB) \cite{olson2017pmlb}, NAS-Bench-360 \cite{DBLP:journals/corr/abs-2110-05668}. There have also been multiple AutoML competitions organized, with the winner being the algorithm that yields the highest performance on a hidden test set, such as the ChaLearn AutoML Challenge Series from 2015-2018 \cite{Guyon2019}. Currently ongoing at the time of writing is also the AutoML Decathlon at the NeurIPS'22 Competition Track \cite{automl-decathlon}.


It is also informative to investigate problems with a significant difference in the performance of the tested algorithms. Such problems may lead to insight for developers into potential areas of improvement and, for users, recommendations on which algorithms to use in a given circumstance. This was the motivation behind the DIverse and GENerative ML Benchmark (DIGEN) which provides forty synthetic datasets with diverse generative functions designed to separate the performance of common machine learning algorithms \cite{doi:10.1126/sciadv.abl4747}. DIGEN provides an excellent opportunity to test how robust AutoML learning algorithms are to a diverse set of problems. DIGEN also provides a Python package, found here \url{https://github.com/EpistasisLab/digen}, with code to benchmark machine learning algorithms and produce summary figures. This paper compares the performance of state-of-the-art AutoML algorithms with individually hyperparameter-tuned machine learning algorithms on these datasets.


\section{Materials and Methods}

This study used four popular AutoML algorithms: TPOT, Auto-Sklearn, Auto-Sklearn 2, and H2O AutoML on a new collection of forty synthetic datasets called DIGEN.  

\subsection{AutoML algorithms}

\paragraph{TPOT} Tree-based Pipeline Optimization Tool (TPOT) utilizes evolutionary algorithms to optimize a tree-shaped pipeline of common machine learning steps. It can also be configured with a template to restrict it to a certain shape and sequence of module categories (grouped by selectors, transformers, and classifiers) \cite{OlsonGECCO2016,olson2018system,Olson2016EvoBio,le2020scaling,pmlr-v64-olson_tpot_2016,10.1093/bioinformatics/btz470}. 

We compared Three different configurations of TPOT, each with a different restriction on the pipeline shape:
\begin{enumerate}
    \item \emph{TPOT\_C} - TPOT restricted to only using a single Classifier. That is, no feature selection or prepossessing steps.
    \item \emph{TPOT\_STC} - TPOT with a template consisting of a feature Selector followed by a feature Transformer ending in a  Classifier.
    \item \emph{TPOT\_Base} - The default TPOT with no template and thus can generate unrestricted pipeline shapes.
\end{enumerate}

\paragraph{Auto-Sklearn} Auto-Sklearn uses Bayesian Optimization and meta-learning to identify best-performing machine learning algorithms, perform hyperparameter optimization, and generate ensembles \cite{feurer-neurips15a}. 

\paragraph{Auto-Sklearn 2} Auto-Sklearn 2 builds upon the meta-learning algorithm with Portfolio Successive Halving and automates more decisions with a Policy Section algorithm. These changes led to increased speeds and performance \cite{feurer-arxiv20a}.

\paragraph{H2O AutoML} H2O AutoML uses random search to identify and train base models, which a meta learner then selects from to combine into a stacked ensemble \cite{H2OAutoML20}.

\subsection{Benchmarking data - DIGEN}

Diverse and Generative ML Benchmark (DIGEN) provides 40 mathematical functions that each map ten input features into a binary classification. These mathematical functions were designed to maximize the diversity of the performance and ranking across several common hyperparameter-tuned machine-learning algorithms. The datasets, therefore, highlight the strengths and weaknesses of the included machine learning algorithms.

In addition to providing the 40 datasets, DIGEN also provides benchmark scores for several Scikit-learn machine learning algorithms after hyperparameter optimization using Optuna. For comparison, in our results, we also include the scores provided by DIGEN for the following Scikit-Learn methods: GradientBoostingClassifier, LGBMClassifier, XGBClassifier, DecisionTreeClassifier, LogisticRegression, KNeighborsClassifier, RandomForestClassifier, SVC, and MLPClassifier \cite{doi:10.1126/sciadv.abl4747}.


\subsection{Experiment design}




\begin{table}[ht!]
 \scriptsize
\centering
\caption{AutoML Algorithms and their parameters.}
\label{tab:ml-params}
\begin{tabular}{|c|l|p{0.5\linewidth}|}
\hline
{\bf Algorithm} & {\bf Parameters} & {\bf Values}\\
\hline


\textbf{TPOT} &  `template' & \textbf{TPOT\_Base} : None \newline \textbf{TPOT\_C} : `Classifier' \newline
\textbf{TPOT\_STC} : `Selector-Transformer-Classifier\\
  &  `population\_size' & 100 \\
  &  `generations' & 100 \\
  &  `scoring'& `roc\_auc' \\
  &  `n\_jobs'& 48 \\
  &  `verbosity' & 2 \\
  &  `cv' & 10 \\
   & `max\_time\_mins' & 1200/60 \\
  \hline
\textbf{Auto-Sklearn} & 
      `n\_jobs' & 48 \\
      &   `metric' & autosklearn.metrics.roc\_auc \\
      &   `resampling\_strategy\_arguments' & \{`cv': 10\} \\
       &  `time\_left\_for\_this\_task' & 1200  \\
       &  `memory\_limit' & 1000000 \\
      \hline
\textbf{Auto-Sklearn 2} & `n\_jobs' & 48 \\
       &  `metric'& autosklearn.metrics.roc\_auc \\
      &   `time\_left\_for\_this\_task'& 1200 \\
       &  `memory\_limit'& 1000000 \\
              \hline
\textbf{H2O AutoML} & 
    `stopping\_metric' & `AUC' \\
   &  `sort\_metric' & `AUC' \\
   &  `nfolds'& 10 \\
   &  `max\_runtime\_secs' & 1200 \\
   & `max\_mem\_size' (h2o.init() ) & `1000G' \\
   &  `n\_jobs' (h2o.init() ) & 48 \\
    \hline
\end{tabular}
\label{table:params}
\end{table}

Using the $evaluate$ function provided by DIGEN \cite{doi:10.1126/sciadv.abl4747}, each AutoML algorithm was fit to an 800-sample training set and evaluated on a 200-sample test set. Parameters were set for each AutoML algorithm to make the experimental setup as similar as possible. All algorithms were set to optimize Area Under the Receiver Operating Curve (AUROC) using 10-fold cross-validation (except for Auto-Sklearn 2, which uses meta-learning to decide the number of folds). All algorithms were set to use 48 threads through the n\_jobs parameter. H2O, Auto-Sklearn 1 and 2 require a memory limit which was set to 1000GB, the full capacity of the compute node. A 20-minute time limit for fitting the model was provided using the appropriate parameter for each model. As some algorithms occasionally went over time, a terminate signal was called after 120\% of the time limit had passed for TPOT and H2O. This signal was not used with Auto-Sklearn as it does not support manual termination via signals. The exact settings of parameters for AutoML algorithms are presented in Table \ref{table:params}.



All models were run on a High Performance Computing cluster node with an Intel Xeon Gold 6342 CPU with 48 threads and 1TB of memory. 





\section{Results}
AutoML algorithms often reach higher ranks on average compared to indi-


Scores for the individual machine learning algorithms optimized with Optuna were taken from the original study \cite{doi:10.1126/sciadv.abl4747}. Figure \ref{fig:box_all_first_rank_auroc_main.pdf} illustrates the distribution of relative rankings of AUROC scores across the 40 datasets for the tested algorithms. AutoML algorithms often reach higher ranks on average compared to individual machine learning algorithms. Note that the performance, and thus the rankings, of AutoML algorithms can and do vary from run to run. Additionally, as most AutoML algorithms perform similarly to each other, as shown in the following figures, the exact rankings here may be less representative of actual performance differences and more so due to negligible differences in scores that only by chance happened to slightly favor one algorithm more often. 


\begin{figure}[htb!]
    \centering
    \centerline{\includegraphics[scale=0.35]{figs/box_all_first_rank_auroc_main.pdf}}
    \caption{Distribution of relative rankings of raw AUROC scores for each dataset.}
    \label{fig:box_all_first_rank_auroc_main.pdf}
\end{figure}




Figure \ref{fig:automl_parallel_coords_from_optuna_colored_by_tpot.pdf} illustrates the difference in test score on each dataset from the best test score of the best Optuna-optimized machine learning methods for one run of each AutoML algorithm. Most AutoML algorithms performed similarly to the best-optimized individual algorithms. On average, TPOT\_STC and Auto-Sklearn 2 performed the best relative to the best individual algorithm scores, with an average difference in AUROC scores of 0.000803 and 0.000801, respectively. Most AutoML AUROC scores are within 0.01 of the best Optuna-optimized model; however, there were occasional cases where AutoML performed better or worse by a more considerable amount. As the best Optuna-optimized models were already at or near-optimal performance, it is expected that we may not be able to see performance gains for most datasets by AutoML algorithms. 

\begin{figure}[htb!]
    \centerline{\includegraphics[scale=0.3]{figs/automl_parallel_coords_from_optuna_colored_by_tpot.pdf}}
    \caption{Parallel coordinates plot illustrating the differences between the test scores of each AutoML algorithm and the best Optuna-optimized individual machine learning algorithm on each dataset. Each horizontal line represents a different dataset.}
    \label{fig:automl_parallel_coords_from_optuna_colored_by_tpot.pdf}
\end{figure}




Next, we look at how often AutoML matches the performance of individually tuned machine-learning methods. Figure \ref{fig:y_better_than_x_by_0_99p_run_0_best_optuna.pdf} illustrates the number of times each algorithm reaches at least 99\% of the performance of the others. Also included in the graph is a category for "Best Optuna," which takes the best test score from each of the individual machine-learning algorithms. This is to visualize how AutoML compares to the best-performing individual algorithm more easily. There are some instances where each AutoML algorithm did not reach the best performance of some Optuna-optimized methods. While AutoML algorithms performed similarly to each other, this figure highlights that some algorithms performed better than others on a few datasets. In other words, no AutoML algorithm performed best on every dataset.  

\begin{figure}[htb!]
    \includegraphics[scale=0.4]{figs/y_better_than_x_by_0_99p_run_0_best_optuna.pdf}
    \caption{A matrix showing wins-loses between the pairs of the methods. Each cell represents the count of datasets where the algorithm in the row reaches at least 99\% of the AUROC score of the algorithm in the column.}
    \label{fig:y_better_than_x_by_0_99p_run_0_best_optuna.pdf}
\end{figure}

While TPOT\_STC has a slightly higher average rank than Auto-Sklearn 2, a closer inspection of the scores shows that they may actually be more similar than implied by the rankings. TPOT\_STC and Auto-Sklearn 2 are within 99\% of each other's scores for 34 out of 40 datasets. As for the remaining datasets, there are three where Auto-Sklearn 2 outperforms and two where TPOT\_STC outperforms. In this respect, they are more or less evenly matched.

\begin{figure}[htb!]
    \centering
    \includegraphics[scale=0.7]{figs/box_std_per_dataset_automl_main_strip.pdf}
    \caption{ Standard deviation of AUROC scores across 10 runs of a single dataset.}
    \label{fig:std_plot}
\end{figure}

Stochasticity in the AutoML algorithms can lead to different final models with different performances in each run. To get a sense of how consistently AutoML algorithms find reasonable solutions to a given dataset across multiple runs, we reran each algorithm for a total of ten times. Each point in Figure \ref{fig:std_plot} illustrates the standard deviation scores calculated from the AUROC scores for ten runs of a single dataset. This graph shows that most algorithms consistently found similarly performing models for most datasets. Some algorithms had a more variable performance on specific datasets. In these cases, the algorithm may have to be run multiple times to find the ideal solution. Auto-Sklearn 2 was the most consistent run to run, followed by Auto-Sklearn and TPOT\_STC. TPOT\_Base, TPOT\_C, and H2O had the most variance in performance. Next, we counted the number of times each AutoML algorithm failed to reach 99\% of the performance of the best Optuna-optimized algorithm for each dataset over ten runs. Figure \ref{fig:cummulative_misses.pdf} plots the cumulative sum of these misses on datasets of increasing difficulty. The step height reflects the number of attempts (up to 10) in which a given AutoML algorithm failed to match the best non-AutoML algorithm for a given dataset. Again, Auto-Sklearn 2 is most consistent at finding high-performing models, followed by TPOT\_STC. TPOT\_C and H2O were the least consistent overall. 


\begin{figure}[htb!]
    \centering
    \includegraphics[scale=0.4]{figs/cummulative_misses.pdf}
    \caption{Cumulative sum of the number of worse performing runs (out of 10) for each dataset. Datasets were sorted from the fewest to the most misses.}
    \label{fig:cummulative_misses.pdf}
\end{figure}

All autoML algorithms struggled the most on digen29\_8322, where an Optuna-optimized XGBoost was often the better model. Both TPOT\_STC and Auto-Sklearn 2 only reached 99\% of the performance of XBGoost once out of ten runs on digen29\_8322. TPOT\_STC was also less consistent on digen1\_6265 and digen3\_769, where it did not reach optimal performance 30\% and 80\% of the time, respectively. Auto-Sklearn 2 and TPOT\_STC also had a few other datasets where it would occasionally (less than 20\% of runs) fail to reach the best individual algorithm score. 


\begin{figure}
    \includegraphics[scale=0.5]{figs/STC_models_all_1200s.pdf}
    \caption{Pipelines generated by TPOT\_STC from all ten runs of each dataset. The left, middle, and right bars respectfully correspond with the selector, transformer, and classifier components. Each color represents a unique pipeline configuration, read from left to right. The width of each color corresponds to the number of times that configuration was used. }
    \label{fig:All_STC_models}
\end{figure}


TPOT can find a diverse set of pipelines to solve the different datasets. For each dataset, TPOT would often find different pipelines run to run. Figure \ref{fig:All_STC_models} illustrates the pipelines found across all ten runs of each dataset. 


\section{Conclusion}

In this study, we demonstrated the ability of AutoML algorithms to consistently find high-performing machine-learning pipelines on a variety of uniquely defined datasets. Of the AutoML algorithms, Auto-Skearn 2 and TPOT\_STC performed best, considering both average score and consistency. 

Of the TPOT templates explored, TPOT\_STC had the best and most consistent performance. TPOT\_C was the most variable. The failure of TPOT\_C to find well-performing models signals that more work needs to be done to improve hyperparameter search within TPOT. Adding the feature selector and transformer to the TPOT template improved the consistency and overall performance. The same selectors and templates are found in TPOT\_Base; however, having an unrestricted pipeline negatively impacted performance. One possible explanation could be that by restricting the search space, TPOT may have spent less time on unnecessarily large models. Future work on TPOT will explore better initialization and more efficient hyperparameter searching, possibly with Optuna, to improve the consistency from run to run. 

While an individual machine learning algorithm optimized with Optuna could sometimes perform better than any of the AutoML algorithms, in general, the AutoML algorithms were more broadly applicable. In particular, digen29\_8322 was the most challenging dataset for all AutoML algorithms. Incidentally, this also happens to be the dataset with the most noise. Several DIGEN datasets execute a forced reclassification step in which some target values of one class are randomly reassigned to the other to yield an exact 50\% split in target classes. For example, in the digen29\_8322 dataset, the generative function leads to 691 ones and 309 zeros. To even the classes, a random set of 191 ones, 19.1\% of the dataset got reclassified to zero. This step could be interpreted as adding noise to the dataset. Fourteen datasets in total have similar directional noise. Of those, 13 have less than 5\% of the labels randomly reassigned, and one, digen29\_8322, has 19.1\% of labels reassigned.


It is important to note that due to differences in testing environments (CPU model, number of threads, time limits, etc.), this paper does not make claims about whether or not the AutoML algorithms found solutions in less time or with fewer models explored as compared to the methodology in the original paper.

One limitation of this study is that the sample size for the test set is relatively small, at only 200 samples. The ranking of the different pairs of models may be misleading as the score differences between ranks may be negligible and/or due to a biased estimate from the small test set size. A different sample of 200 test points is likely to change the scores around the differences we observe. Therefore, it is difficult to precisely conclude whether or not one algorithm is an overall better or worse choice from this study. Additionally, the original DIGEN benchmarking on the individually tuned machine learning algorithms included in this study used the Scikit-learn "predict(X)" function rather than "predict\_proba(X)" or "decision\_function(X)" as used by the AutoML algorithms when available in the cross-validation loop. As this affects the AUROC score, it is possible that this difference also affects the overall training and changes the final hyperparameters tuned by Optuna.


As the DIGEN datasets are relatively simple and do not require heavy feature selection or transformation, it is unclear how the results from these experiments would generalize AutoML recommendations to other real-world problems. While some algorithms performed similarly in this benchmark, there may be situations where one would be preferred over others. More complex benchmarks will be required to tease apart the strengths and weaknesses of different AutoML algorithms. Datasets could also be designed to target a benchmark toward specific use cases. For example, one area of interest is heterogeneity which could be simulated by combining DIGEN equations to simulate two or more different population groups. This setup could test the ability of AutoML to simultaneously identify population groups and predict the outcome given that group. Additionally, it would be important to consider different combinations of sample size, the number of features, and generative model complexity, which could provide insight into the potential differences of AutoML algorithms under different scenarios.

Code is available in the following GitHub repository \url{https://github.com/perib/automl_digen_benchmark}

\section{Acknowledgement}

We would like to acknowledge funding from National Institutes of Health (NIH) grants U01 AG066833 and R01 LM010098.

