46. The computer-readable storage medium of claim 43 , wherein the operations executed comprise: receiving a plurality of votes cast by human searchers, and notifying the human searchers to vote in an order based on availability of the human searchers.
43. A computer-readable storage medium having a program recorded therein for causing a computer to execute operations improving relevance of search results, comprising: determining whether an unranked review item has been associated with a reference item; receiving a vote of a human searcher regarding relevance of a ranked review item and the unranked review item to a reference item presented to the human searcher; selecting the human searcher based on a ranking of the human searcher associated with the reference item; modifying a ranking of the ranked review item for the reference item based on the vote; and presenting the review items responsive to a search request matching the reference item in an order based on the ranking.