In this work, a principled, data-driven strategy to assess the transferability of benchmarking results and to construct more challenging and/or representative datasets will be presented, based on active learning. With a particular focus on the simulation of pericyclic reactions, we train here a machine learning model on a small pre-existing benchmarking dataset, to identify regions of chemical space for which activation energies computed with different functional approximations diverge the most. Next, a Bayesian optimization (BO) algorithm is applied to identify the reactions in the chemical space surrounding the initial dataset exhibiting the biggest variations in DFT computed activation energies. The selected reactions then ought to be representative of more challenging patches of chemical space than those present in the training set.