10. The scheduler system of claim 7 , wherein the scheduler is configured to determine the content change frequency from a content checksum stored in a history log.
7. A scheduler system for a search engine crawler, comprising: a computer; a history log containing document identifiers corresponding to documents on a network previously indexed by the search engine crawler, wherein each document identifier has a corresponding document; and a scheduler, executed by the computer, and configured to process each document identifier in a set of the document identifiers in the history log by determining a query-independent score indicative of a page rank of the corresponding document relative to other documents on the network, determining a content change frequency of the document corresponding to the document identifier by comparing information stored for successive downloads of the corresponding document, determining an age of the corresponding document, wherein the age is associated with the time of the last download of the corresponding document by the crawler, determining a first score for the document identifier that is a function of the query-independent score and the determined content change frequency and the determined age of the corresponding document, comparing the first score against a threshold value, and conditionally scheduling the corresponding document for indexing based on the results of the comparison, wherein the history log and scheduler are stored on computer-readable media.