1. A computer-implemented method of training a classifier, such that the trained classifier is configured to map an instance to one of a plurality of classes, comprising: providing a query space, wherein each of a plurality of queries of the query space is associated with a subset of a plurality of instances and a subset of a plurality of features, where each of the plurality of queries of the query space includes a relevant characteristic function that describes, for that query, a function of instance-feature values associated with that query and of instance-class probabilities associated with that query, wherein the instance-class probabilities are an indication of a probabilistic model of mapping of the instances associated with the query to at least one of the plurality of classes, the query space tangibly embodied in a computer-readable medium; for each of the plurality of queries, applying a query utility function to determine a query utility value for that query; based, at least in part, upon the query utility value determined for each of the plurality of queries, identifying one or more queries of the plurality of queries to be presented for editorial feedback; operating a computing device to present the identified queries and receive an indication of commentary from at least one editor for each of the identified one or more queries of the plurality of queries; maintaining a classifier framework tangibly embodied in a computer-readable medium, the classifier framework configured to provide class probabilities for the instances according to a tunable parameter vector; operating a computing device to determine a distortion value for the identified queries by applying a distortion function to a deviation of the classifier framework response from the indication of editorial commentary for that query; and operating a computing device to adjust the tunable parameter vector based on a cost function that considers a regularization component and the distortion values over the queries for which the editors gave commentary.
1. A computer-implemented method of training a classifier, such that the trained classifier is configured to map an instance to one of a plurality of classes, comprising: providing a query space, wherein each of a plurality of queries of the query space is associated with a subset of a plurality of instances and a subset of a plurality of features, where each of the plurality of queries of the query space includes a relevant characteristic function that describes, for that query, a function of instance-feature values associated with that query and of instance-class probabilities associated with that query, wherein the instance-class probabilities are an indication of a probabilistic model of mapping of the instances associated with the query to at least one of the plurality of classes, the query space tangibly embodied in a computer-readable medium; for each of the plurality of queries, applying a query utility function to determine a query utility value for that query; based, at least in part, upon the query utility value determined for each of the plurality of queries, identifying one or more queries of the plurality of queries to be presented for editorial feedback; operating a computing device to present the identified queries and receive an indication of commentary from at least one editor for each of the identified one or more queries of the plurality of queries; maintaining a classifier framework tangibly embodied in a computer-readable medium, the classifier framework configured to provide class probabilities for the instances according to a tunable parameter vector; operating a computing device to determine a distortion value for the identified queries by applying a distortion function to a deviation of the classifier framework response from the indication of editorial commentary for that query; and operating a computing device to adjust the tunable parameter vector based on a cost function that considers a regularization component and the distortion values over the queries for which the editors gave commentary. 6. The method of claim 1 wherein: the distortion value for each of at least some of the queries is weighted by a relative importance score for that query; and the relative importance score of a query is the same for all queries which share the same relevant characteristic function.