A primary consideration when developing machine learned models is determining an appropriate set of hyperparameters. For an MLP there are hyperparameters for the model, such as the network architecture of NNPs, as well as hyperparameters for the descriptor, for example the radial cutoff. Active learning has its own hyperparameters, including the batch size and uncertainty threshold, and also a query strategy. In order to apply active learning efficiently with CSP datasets we first investigated optimising the hyperparameters and selection strategy. The aim was to identify the approach that yields the smallest training set that accurately captures the whole landscape, as measured by small test errors, and does so consistently with minimal variation. The dataset we chose for these studies was a predicted landscape for oxalic acid containing 1965 crystal structures. We chose this landscape because oxalic acid is a known challenging system for empirical force fields and thus learning either the total energy or ∆-learning the correction from force field to DFT will be meaningful tests. Furthermore, the small size of the oxalic acid structures meant calculating the entire dataset at the target level, PBE-D3, was possible, and thus results were able to be verified through 5-fold cross-validation.