Advantage of active learning on classification tasks

Question

I am looking for a specific type of experimental test of active learning. Given some artificial or natural learning task that consists of classifying inputs $x$ from a large input space $X$. There is some underlying function $f: X \rightarrow \{0,1\}$ to be learned, this function can come from some restricted function class $C$. The subject knows the description of the function class, but has to learn the specific parameters that define the function. I am looking for experiments that had two conditions of this type:

1. The subject is given labeled samples $(x,f(x))$ from some distribution on $X$ that they have no control over.
2. The subject is allowed to ask for specific $x \in X$ and receive the labeled sample $(x,f(x))$.

The first condition would correspond to passive learning, while the second to active learning.

From arm-chair expectations (or formal theories like CoLT), one would expect that on certain tasks the subjects in the active condition would be able to significantly outperform (predict the underlying function more accurately) than in the passive condition.

Can you provide references to specific experiments that would fit in this paradigm? What were their results? On what sort of tasks does active learning provide an advantage? How significant is this advantage?

---

Example

An example would be a subject sitting in front of a blank screen. They are told "there is a straight line that divides this screen into a red and green region. Can you find it?". Thus, the set $X$ would correspond to coordinates, and an input $x$ would be a specific coordinate, say $x = (1,3)$. The function class would be the set of straight line partitions of the screen. A specific function with $0$ meaning red and $1$ - green might be:

$$f((a,b)) = \begin{cases}0 & \mathrm{if} \; a > b \\ 1 & \mathrm{otherwise}\end{cases}$$

In the active task, the participant can click on any point $x$ on the screen to turn it from blank to either red (if $f(x) = 0$) or green (if $f(x) = 1$). In the passive trial, some number of points are selected and revealed at random to the subject.

Answer 1

This isn't my area of expertise, but it's an interesting question so I did some poking around and ran across a couple of relevant papers.

Human active learning

http://papers.nips.cc/paper/3456-human-active-learning

Castro, R. M., Kalish, C., Nowak, R., Qian, R., Rogers, T., & Zhu, X. (2009). Human active learning. In Advances in neural information processing systems (pp. 241-248).

In this paper, the authors compare human performance on a classification task using (a) randomly selected samples, (b) human-selected examples (i.e., "active learning"), and (c) machine-selected samples ("machine-assisted active learning"). Participants in all three conditions were tested regularly on the classification task as they accumulated examples. Somewhat surprisingly (for me -- I would have expected the human-selected condition to learn the fastest), participants in the machine-selected example case were the fastest learners, particularly when label noise was large.

Category learning through active sampling

http://palm.mindmodeling.org/cogsci2010/papers/0041/paper0041.pdf

Markant, D., & Gureckis, T. M. (2010). Category learning through active sampling. In Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 248-253).

The task in this paper is a linear separation-boundary location task like the example you provide. It might be worth a look just to see whether it is similar to what you were describing.

(Also here's a more recent journal-length paper from the same authors: Markant, D. B., & Gureckis, T. M. (2014). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94.)