16. The system of claim 2 , wherein the crawl ordering engine orders the web pages to fetch by the web crawler by using the combined query-based estimate and query-independent estimate for said particular web page and computing an expected impact score using query-independent features of the particular uncrawled web page.
2. The system of claim 1 , wherein the crawling policy manager for estimating the impact of fetching the particular uncrawled web page further combines a query-based estimate and a query-independent estimate for said particular uncrawled web page by computing a weighted average of said query-based estimate and said query-independent estimate for said particular uncrawled web page; wherein said query-based estimate for said particular uncrawled web page indicates how well said particular uncrawled web page meets neediness and relevance requirements of the needy queries and is determined based, at least in part, on the anchortext and the URL that point to the particular uncrawled web page; wherein said query-independent estimate of said particular uncrawled web page is determined based on query-independent features of said particular uncrawled web page.