6. A method of scheduling a search engine crawler using a scheduler system, comprising: at a server with one or more processors and memory, and one or more programs stored in the memory that execute on the one or more processors: selecting a first subset of document identifiers from a set of document identifiers corresponding to documents on a network; computing priority scores for the first subset of document identifiers; forming a second subset of document identifiers based on the priority scores and status data collected during one or more previous crawls by the search engine crawler and by removing from the first subset one or more document identifiers identified as unreachable in a plurality of prior crawls or associated with download errors in a plurality of prior crawls; scheduling for crawling the second subset of document identifiers; selecting one or more document identifiers from the second subset whose corresponding documents will be reused based on historical status data and retrieving and indexing documents corresponding to the selected document identifiers, wherein the documents are retrieved from a repository distinct from web hosts corresponding to the document identifiers.
6. A method of scheduling a search engine crawler using a scheduler system, comprising: at a server with one or more processors and memory, and one or more programs stored in the memory that execute on the one or more processors: selecting a first subset of document identifiers from a set of document identifiers corresponding to documents on a network; computing priority scores for the first subset of document identifiers; forming a second subset of document identifiers based on the priority scores and status data collected during one or more previous crawls by the search engine crawler and by removing from the first subset one or more document identifiers identified as unreachable in a plurality of prior crawls or associated with download errors in a plurality of prior crawls; scheduling for crawling the second subset of document identifiers; selecting one or more document identifiers from the second subset whose corresponding documents will be reused based on historical status data and retrieving and indexing documents corresponding to the selected document identifiers, wherein the documents are retrieved from a repository distinct from web hosts corresponding to the document identifiers. 7. The method of claim 6 , further comprising: applying at least one exception filter to at least a plurality of the document identifiers to identify document identifiers to be excluded from a next crawl.