14. The system as in claim 9 , wherein the first probability depends on one of more of: source parameters, a type of the URL, an index size and a workload of a web crawler.
9. An adaptive feedback system for ensuring timeliness of a collection of web pages, the system comprising: a query processor that extracts a uniform resource locator (URL) from a search result; a cache including a process for determining whether or not a web page corresponding to the URL is present in whole or in part in the collection; a web crawler coupled to the processor wherein: when the web page is determined to be present in the collection, the web crawler downloads and replaces the web page in the collection in accordance with a first probability, wherein the first probability depends on an age of the web page in the collection; and when the web page is determined not to be present in the collection, the web crawler downloads and including the web page in the collection, wherein the processor extracts hyperlinks from the web page corresponding to the URL and directs the web crawler to download the web pages corresponding to the hyperlinks each with a second probability, wherein the second probability depends on the number of hyperlinks in the web page.