1. A method of crawling the Web, said method comprising: crawling Web pages on the Web starting from a given set of seed Universal Resource Locators (URLs); partitioning crawled Web pages into sets of relevant and irrelevant pages; discovering from said sets of relevant and irrelevant pages a set of exclusion and inclusion patterns; and restricting subsequent crawling of the Web through said set of exclusion and inclusion patterns.