14. A computer program product comprising at least one non-transitory computer readable medium having computer program instructions tangibly embodied thereon, the computer program instructions to configure at least one computing device to train a classifier such that the trained classifier is configured to map an instance to one of a plurality of classes, the computer program instructions, when executed by a processor, causing the processor to perform steps, comprising: determining, from a plurality of instances that each have corresponding features of a plurality of features, clusters of instances that are homogeneous with respect to dependency of instance class assignments on the instance-feature values; constructing a reduced query space from a query space based on the determined clusters, the query space including a plurality of queries, each of the plurality of queries being associated with a subset of the plurality of instances and a subset of the plurality of features, wherein the reduced query space includes a second plurality of queries, the second plurality of queries including a subset of the plurality of queries; approximating a query utility function using the determined clusters; for each of the second plurality of queries, applying the query utility function to determine a query utility value for that query; based, at least in part, upon the query utility value determined for each of the second plurality of queries, identifying one or more queries of the second plurality of queries to be presented for editorial feedback; operate a computing device to present the identified queries and receive an indication of commentary from at least one editor for each of the identified queries; maintain a classifier framework tangibly embodied in a computer-readable medium, the classifier framework configured to provide class probabilities for the instances according to a tunable parameter vector; determine a distortion value for each of the identified queries by applying a distortion function to a deviation of the classifier framework response from the indication of editorial commentary for that query; and adjust the tunable parameter vector based on a cost function that considers a regularization component and the distortion values over the queries for which the editors gave commentary.
14. A computer program product comprising at least one non-transitory computer readable medium having computer program instructions tangibly embodied thereon, the computer program instructions to configure at least one computing device to train a classifier such that the trained classifier is configured to map an instance to one of a plurality of classes, the computer program instructions, when executed by a processor, causing the processor to perform steps, comprising: determining, from a plurality of instances that each have corresponding features of a plurality of features, clusters of instances that are homogeneous with respect to dependency of instance class assignments on the instance-feature values; constructing a reduced query space from a query space based on the determined clusters, the query space including a plurality of queries, each of the plurality of queries being associated with a subset of the plurality of instances and a subset of the plurality of features, wherein the reduced query space includes a second plurality of queries, the second plurality of queries including a subset of the plurality of queries; approximating a query utility function using the determined clusters; for each of the second plurality of queries, applying the query utility function to determine a query utility value for that query; based, at least in part, upon the query utility value determined for each of the second plurality of queries, identifying one or more queries of the second plurality of queries to be presented for editorial feedback; operate a computing device to present the identified queries and receive an indication of commentary from at least one editor for each of the identified queries; maintain a classifier framework tangibly embodied in a computer-readable medium, the classifier framework configured to provide class probabilities for the instances according to a tunable parameter vector; determine a distortion value for each of the identified queries by applying a distortion function to a deviation of the classifier framework response from the indication of editorial commentary for that query; and adjust the tunable parameter vector based on a cost function that considers a regularization component and the distortion values over the queries for which the editors gave commentary. 20. The computer program product of claim 14 , wherein applying the utility function is not performed for queries in the plurality of queries that are not in the second plurality of queries.