Batch-mode active learning: How to select the batch that will bring the largest model improvement I am facing a problem where I want to use active learning to improve my classifier. Basically, I can choose data from one (and only one) data set among a set of candidate data sets. The question is which one to choose? 

In other words, given a set of candidate data sets that I can use improve my classification model, which one is going to improve the model most?
Can I use the some metric (eg., average) from the objective functions of the batches? Do I need to normalise the objectives across all data sets? Have metrics to infer the magnitude of model improvement been proposed?

At this stage, my objective function looks like: 

a  Uncertainty + (1-a) Diversity


where a is a weight factor, Uncertainty is the uncertainty in the model prediction for a given data point and Diversity is a measure of distance between the data point and the training population.

Any help would be greatly appreciated.

Cheers

Franz
Your question is actually the whole point of active learning. You probably need to read about existing approaches in active learning in order to find the one which suits your needs.

I'm not up to date at all on the topic but a traditional approach was to train several models on available data, make them predict on all the unannotated instances then use majority voting: instances for which the models tend to agree are "easy" to predict, whereas those for which models make different predictions are "hard" so potentially more valuable to improve performance.