1. A scheduler system for a search engine crawler, comprising: a memory for storing a set of document identifiers corresponding to documents on a network and associated status data collected during one or more previous crawls by the search engine crawler; and a plurality of schedulers configured to select a subset of the document identifiers for crawling, at least some of the schedulers configured to compute priority scores for the subset of document identifiers and to schedule for crawling at least a portion of the subset of document identifiers based on the priority scores and status data; wherein the scheduling includes removing from the subset one or more document identifiers that were unreachable in a plurality of consecutive prior crawls or had download errors in a plurality of consecutive prior crawls; and wherein at least some of the schedulers are configured to: select one or more document identifiers whose corresponding documents will be reused based on historical status data; and retrieve and index documents corresponding to the selected document identifiers, wherein the documents are retrieved from a repository distinct from web hosts corresponding to the document identifiers.
1. A scheduler system for a search engine crawler, comprising: a memory for storing a set of document identifiers corresponding to documents on a network and associated status data collected during one or more previous crawls by the search engine crawler; and a plurality of schedulers configured to select a subset of the document identifiers for crawling, at least some of the schedulers configured to compute priority scores for the subset of document identifiers and to schedule for crawling at least a portion of the subset of document identifiers based on the priority scores and status data; wherein the scheduling includes removing from the subset one or more document identifiers that were unreachable in a plurality of consecutive prior crawls or had download errors in a plurality of consecutive prior crawls; and wherein at least some of the schedulers are configured to: select one or more document identifiers whose corresponding documents will be reused based on historical status data; and retrieve and index documents corresponding to the selected document identifiers, wherein the documents are retrieved from a repository distinct from web hosts corresponding to the document identifiers. 5. The system of claim 1 , wherein the priority scores are calculated using a priority function given by 
 Priority Score i =page_importance i *boost factor, where the subscript i denotes an ith document identifier and the boost factor is a scalar that is used to demote or promote the priority score of the document identifier.