1. A method that is executed by a computer processor on a computing device, the method comprising: receiving a first search result list returned by a first ranker system responsive to a search query; receiving a second search result list returned by a second ranker system responsive to the search query; employing a first classifier to predict whether a search preference judgment of the first search result list and the second search result list indicates a quality difference between the first search result list and the second search result list; when the first search result list and the second search result list for the search query are predicted to have the quality difference, determining a confidence level for whether the search preference judgment indicates one of the first search result list or the second search result list is preferred; inserting the search query, the first search result list, and the second search result list in a first set of search queries and search result list pairs to be manually judged for search preference judgments when: the first search result list and the second search result list for the search query are predicted to have the quality difference; and the confidence level is equal to or below a threshold value; wherein the search query, the first search result list, and the second search result list are otherwise excluded from the first set of the search queries and the search result list pairs to be manually judged for search preference judgments; and transmitting the first set of the search queries and the search result list pairs to one or more annotator computing devices for manual judgment.