1. A computer-implemented method comprising the steps of: receiving, at a search engine operating on one or more computing devices from a traffic server that is not the search engine and is not operating on the one or more computing devices, network usage information that indicates a frequency with which users of a plurality of users requested, during a particular period of time, a particular document through the traffic server, which monitors frequencies with which documents in a plurality of documents are provided, through the traffic server, from a plurality of origin servers upon which the plurality of documents reside, to the plurality of users in response to a plurality of document-identifying requests from the plurality of users; the search engine positioning the particular document within a queue at a queue position that is selected based on said network usage information; the search engine performing a crawling or indexing operation based at least in part on the position of the particular document within the queue.
1. A computer-implemented method comprising the steps of: receiving, at a search engine operating on one or more computing devices from a traffic server that is not the search engine and is not operating on the one or more computing devices, network usage information that indicates a frequency with which users of a plurality of users requested, during a particular period of time, a particular document through the traffic server, which monitors frequencies with which documents in a plurality of documents are provided, through the traffic server, from a plurality of origin servers upon which the plurality of documents reside, to the plurality of users in response to a plurality of document-identifying requests from the plurality of users; the search engine positioning the particular document within a queue at a queue position that is selected based on said network usage information; the search engine performing a crawling or indexing operation based at least in part on the position of the particular document within the queue. 2. The method of claim 1 , further comprising, after crawling said particular document, updating an index used by the search engine based on new content of said particular document.