Validation of exchange-correlation functionals is an essential task in (computational) chemistry, and the design of benchmark datasets is key to this end. In this work, a new approach, based on Bayesian optimization and active learning, for generating more diverse, unbiased, benchmarking datasets is presented. Starting from an initial model trained on 140 data points, our strategy enables the identification of challenging regions of chemical space within only a handful of iterations. Sampling new data points in these regions reveals that we have successfully pushed the mean errors for most functionals higher, and a completely different picture of the performance is obtained, with the ranking of some functionals being changed dramatically. It is important to note that the strategy followed in this paper is easily extensible to other relevant chemical properties (as well as functionals), and is extremely data-efficient.