Advantage of active learning on classification tasks
This isn't my area of expertise, but it's an interesting question so I did some poking around and ran across a couple of relevant papers.

Human active learning

http://papers.nips.cc/paper/3456-human-active-learning

Castro, R. M., Kalish, C., Nowak, R., Qian, R., Rogers, T., &amp; Zhu, X. (2009). Human active learning. In Advances in neural information processing systems (pp. 241-248).

In this paper, the authors compare human performance on a classification task using (a) randomly selected samples, (b) human-selected examples (i.e., "active learning"), and (c) machine-selected samples ("machine-assisted active learning"). Participants in all three conditions were tested regularly on the classification task as they accumulated examples. Somewhat surprisingly (for me -- I would have expected the human-selected condition to learn the fastest), participants in the machine-selected example case were the fastest learners, particularly when label noise was large.

Category learning through active sampling

http://palm.mindmodeling.org/cogsci2010/papers/0041/paper0041.pdf

Markant, D., &amp; Gureckis, T. M. (2010). Category learning through active sampling. In Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 248-253).

The task in this paper is a linear separation-boundary location task like the example you provide. It might be worth a look just to see whether it is similar to what you were describing.

(Also here's a more recent journal-length paper from the same authors: Markant, D. B., &amp; Gureckis, T. M. (2014). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94.)