15. A non-transitory computer-readable medium having stored thereon instructions, which when executed by a processor in a scheduler system for a search engine crawler, cause the processor to perform the operations of: selecting a first subset of document identifiers from a set of document identifiers corresponding to documents on a network; computing priority scores for the first subset of document identifiers; forming a second subset of document identifiers based on the priority scores and status data collected during one or more previous crawls by the search engine crawler and by removing from the first subset one or more document identifiers identified as unreachable in a plurality of prior crawls or associated with download errors in a plurality of prior crawls; scheduling for crawling the second subset of document identifiers; identifying one or more document identifiers from the second subset whose corresponding documents will be reused based on historical status data and retrieving documents corresponding to the identified document identifiers, wherein the documents are retrieved from a repository distinct from web hosts corresponding to the document identifiers.
15. A non-transitory computer-readable medium having stored thereon instructions, which when executed by a processor in a scheduler system for a search engine crawler, cause the processor to perform the operations of: selecting a first subset of document identifiers from a set of document identifiers corresponding to documents on a network; computing priority scores for the first subset of document identifiers; forming a second subset of document identifiers based on the priority scores and status data collected during one or more previous crawls by the search engine crawler and by removing from the first subset one or more document identifiers identified as unreachable in a plurality of prior crawls or associated with download errors in a plurality of prior crawls; scheduling for crawling the second subset of document identifiers; identifying one or more document identifiers from the second subset whose corresponding documents will be reused based on historical status data and retrieving documents corresponding to the identified document identifiers, wherein the documents are retrieved from a repository distinct from web hosts corresponding to the document identifiers. 19. The computer-readable medium of claim 15 , further comprising instructions for: sorting the first subset of document identifiers by priority scores; and scheduling a portion of the sorted document identifiers to be crawled.