8. A system for scheduling documents for crawling, comprising: one or more processors; and memory storing one or more modules; the one or more modules including instructions for: storing sitemap information for a plurality of websites, wherein the information includes a predicted update period for at least a plurality of documents identified by the sitemap information; analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date; updating the stored sitemap information for the identified respective website by downloading updated sitemap information for the identified respective website; and scheduling documents for crawling in accordance with the updated stored sitemap information for the identified respective website.
8. A system for scheduling documents for crawling, comprising: one or more processors; and memory storing one or more modules; the one or more modules including instructions for: storing sitemap information for a plurality of websites, wherein the information includes a predicted update period for at least a plurality of documents identified by the sitemap information; analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date; updating the stored sitemap information for the identified respective website by downloading updated sitemap information for the identified respective website; and scheduling documents for crawling in accordance with the updated stored sitemap information for the identified respective website. 9. The system of claim 8 , wherein the instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date comprise instructions for analyzing the stored sitemap information to identify a respective website having sitemap information that is older than a shortest predicted update period for at least one document listed in the sitemap information for the respective website.