Abstract This paper considers the challenge of evaluating a set of classi ers, asdone in shared task evaluations like the KDD Cup or NIST TREC, without ex-pert labels. While expert labels provide the traditional cornerstone for evaluatingstatistical learners, limited or expensive access to experts represents a practicalbottleneck. Instead, we seek methodology for estimating performance of the clas-si ers (relative and absolute) which is more scalable than expert labeling yet pre-serves high correlation with evaluation based on expert labels. We consider both:1) using only labels automatically generated by the classi ers themselves (blindevaluation); and 2) using labels obtained via crowdsourcing. While crowdsourcingmethods are lauded for scalability, using such data for evaluation raises seriousconcerns given the prevalence of label noise. In regard to blind evaluation, twobroad strategies are investigated: combine & score and score & combine. Combine& Score methods infer a single -gold" label set by aggregating classi erlabels; classi ers are then evaluated based on this single pseudo-gold label set. Onthe other hand, score & combine methods: i) sample multiple label sets from clas-si er outputs, ii) evaluate classi ers on each label set, and iii) average classi erperformance across label sets. When additional crowd labels are also collected,we investigate two alternative avenues for exploiting them: 1) direct evaluation ofclassi ers; or 2) supervision of combine-and-score methods. To assess generality ofour techniques, classi er performance is measured using four common classi cationmetrics, with statistical signi cance tests establishing relative performance of theclassi ers for each metric. Finally, we measure both score and rank correlations be-tween estimated classi er performance vs. actual performance according to expertjudgments. Rigorous evaluation of classi ers from the TREC 2011 CrowdsourcingTrack shows reliable evaluation can be achieved without reliance on expert labels.
@cite_30: In this paper we present some new methods of ranking information retrieval systems without relevance judgement. The common ground of these methods is using a measure we called reference count. An extensive experimentation was conducted to evaluate the effectiveness of the proposed methods using various different standards Information Retrieval evaluation measures for the ranking, like average precision, R-precision, and precision and different document levels. We also compared the effectiveness of the proposed methods with the method proposed by The experimental results showed that the proposed methods are effective, and in many cases are more effective than Soboroff at al.'s method.
@cite_18: How to reduce the amount of relevance judgments is an important issue in retrieval evaluation. In this paper, we propose a novel method using global statistics to rank retrieval systems without relevance judgments. In our method, a series of global statistics of a system, which indicate the percentage of its documents found by k out of all the N systems (k = 1, 2, ..., N), are selected, then a linear combination of the series of global statistics is utilized to fit the mean average precision (MAP) of the retrieval system. Optimal coefficients are obtained by linear regression. No human relevance judgments are required in the entire process. Compared with existing evaluation methods without relevance judgments, our method has two advantages. Firstly, it outperforms all early attempts. Secondly, it is adjustable for different effectiveness measurements, e.g. MAP, precision at n, and so forth.
@cite_3: The most prevalent experimental methodology for comparing the effectiveness of information retrieval systems requires a test collection, composed of a set of documents, a set of query topics, and a set of relevance judgments indicating which documents are relevant to which topics. It is well known that relevance judgments are not infallible, but recent retrospective investigation into results from the Text REtrieval Conference (TREC) has shown that differences in human judgments of relevance do not affect the relative measured performance of retrieval systems. Based on this result, we propose and describe the initial results of a new evaluation methodology which replaces human relevance judgments with a randomly selected mapping of documents to topics which we refer to as pseudo-relevance judgments .Rankings of systems with our methodology correlate positively with official TREC rankings, although the performance of the top systems is not predicted well. The correlations are stable over a variety of pool depths and sampling techniques. With improvements, such a methodology could be useful in evaluating systems such as World-Wide Web search engines, where the set of documents changes too often to make traditional collection construction techniques practical.
@cite_0: Soboroff, Nicholas and Cahan recently proposed a method for evaluating the performance of retrieval systems without relevance judgments. They demonstrated that the system evaluations produced by their methodology are correlated with actual evaluations using relevance judgments in the TREC competition. In this work, we propose an explanation for this phenomenon. We devise a simple measure for quantifying the similarity of retrieval systems by assessing the similarity of their retrieved results. Then, given a collection of retrieval systems and their retrieved results, we use this measure to assess the average similarity of a system to the other systems in the collection. We demonstrate that evaluating retrieval systems according to average similarity yields results quite similar to the methodology proposed by Soboroff et al., and we further demonstrate that these two techniques are in fact highly correlated. Thus, the techniques are effectively evaluating and ranking retrieval systems by "popularity" as opposed to "performance.
@cite_27: "Data fusion" refers to the problem in information retrieval (IR) where several lists of documents ranked against a query are to be merged into a single ranked list for presentation to a user. Data fusion is also known as "metasearch." In a digital library setting data fusion may support operations such as federated search based on multiple repository representations. This paper presents a novel approach to the fusion problem: generative model-based Metasearch (GeM). We suggest viewing the appearance of documents in a return set as the outcome of a probabilistic process; some documents are likely to occur in the model, while others are unlikely. Using Bayesian parameter estimation to fit a multinomial distribution based on the return sets to be merged, GeM achieves a final ranking by listing documents in decreasing probability of generation under the induced model. We also introduce what we call "the impatient reader" approach to normalizing document ranks in service to the fusion operation. We report results from several experiments on TREC data suggesting that GeM, informed with impatient reader document scores, operates at state-of-the-art levels of effectiveness.
@cite_23: This paper addresses the problem of how to rank retrieval systems without the need for human relevance judgments, which are very resource intensive to obtain. Using TREC 3, 6, 7 and 8 data, it is shown how the overlap structure between the search results of multiple systems can be used to infer relative performance differences. In particular, the overlap structures for random groupings of five systems are computed, so that each system is selected an equal number of times. It is shown that the average percentage of a system's documents that are only found by it and no other systems is strongly and negatively correlated with its retrieval performance effectiveness, such as its mean average precision or precision at 1000. The presented method uses the degree of consensus or agreement a retrieval system can generate to infer its quality. This paper also addresses the question of how many documents in a ranked list need to be examined to be able to rank the systems. It is shown that the overlap structure of the top 50 documents can be used to rank the systems, often producing the best results. The presented method significantly improves upon previous attempts to rank retrieval systems without the need for human relevance judgments. This ''structure of overlap'' method can be of value to communities that need to identify the best experts or rank them, but do not have the resources to evaluate the experts' recommendations, since it does not require knowledge about the domain being searched or the information being requested.
@cite_31: Ranking a set retrieval systems according to their retrieval effectiveness without relying on relevance judgments was first explored by [13]. Over the years, a number of alternative approaches have been proposed, all of which have been evaluated on early TREC test collections. In this work, we perform a wider analysis of system ranking estimation methods on sixteen TREC data sets which cover more tasks and corpora than previously. Our analysis reveals that the performance of system ranking estimation approaches varies across topics. This observation motivates the hypothesis that the performance of such methods can be improved by selecting the “right” subset of topics from a topic set. We show that using topic subsets improves the performance of automatic system ranking methods by 26 on average, with a maximum of 60 . We also observe that the commonly experienced problem of underestimating the performance of the best systems is data set dependent and not inherent to system ranking estimation. These findings support the case for automatic system evaluation and motivate further research.
@cite_16: Abstract Test collections have traditionally been used by information retrieval researchers to improve their retrieval strategies. To be viable as a laboratory tool, a collection must reliably rank different retrieval variants according to their true effectiveness. In particular, the relative effectiveness of two retrieval strategies should be insensitive to modest changes in the relevant document set since individual relevance assessments are known to vary widely. The test collections developed in the TREC workshops have become the collections of choice in the retrieval research community. To verify their reliability, NIST investigated the effect changes in the relevance assessments have on the evaluation of retrieval results. Very high correlations were found among the rankings of systems produced using different relevance judgment sets. The high correlations indicate that the comparative evaluation of retrieval performance is stable despite substantial differences in relevance judgments, and thus reaffirm the use of the TREC collections as laboratory tools.
@cite_12: Ranking information retrieval (IR) systems with respect to their effectiveness is a crucial operation during IR evaluation, as well as during data fusion. This article offers a novel method of approaching the system-ranking problem, based on the widely studied idea of polyrepresentation. The principle of polyrepresentation suggests that a single information need can be represented by many query articulations–what we call query aspects. By skimming the top k (where k is small) documents retrieved by a single system for multiple query aspects, we collect a set of documents that are likely to be relevant to a given test topic. Labeling these skimmed documents as putatively relevant lets us build pseudorelevance judgments without undue human intervention. We report experiments where using these pseudorelevance judgments delivers a rank ordering of IR systems that correlates highly with rankings based on human relevance judgments. © 2010 Wiley Periodicals, Inc.
A broad concern with human labeling, such as relevance judging, is ensuring label consistency such that systems can be effectively trained and evaluated. A decade ago, Voorhees showed that reliable evaluation of search systems could be achieved despite significant variations in the underlying relevance judgments @cite_16 . @cite_3 took this idea further: could we forgo human labeling entirely (i.e., blind evaluation) by sampling relevant documents randomly after pooling outputs from all systems participating in a shared task evaluation? While such evaluation indeed correlated positively and significantly with use of human judgments, predicted performance was less reliable for the best systems. Aslam and Savell @cite_0 achieved comparable correlation by simply scoring each system by its mean Jaccard Coefficient over the set of retrieved documents vs. those retrieved by each other system. Wu and Crestani exploited similar reference count'' popularity, modeling expected correlation between relevance and the rank and frequency at which documents are retrieved by systems @cite_30 , similar to rank fusion @cite_27 . Several other blind evaluation methods have also been explored @cite_12 @cite_31 @cite_18 @cite_23 .