17. A computer program product for optimized web domains classification based on progressive crawling with clustering, the computer program product being embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for: crawling a domain to collect data for a subset of pages of a corpus of content associated with the domain; classifying each of the crawled pages into one or more category clusters, wherein the category clusters represent a content categorization of the corpus of content associated with the domain, and wherein the classifying of the each of the crawled pages into the one or more category clusters comprises: determining a category for the each of the crawled pages in the domain; grouping more than one page having the same category into a first cluster; determining whether a number of the more than one page of the first cluster exceeds a first threshold; and in the event that the number of the more than one page of the first cluster does not exceed the first threshold, selecting a new page within the domain to crawl and classify; and determining which of the one or more category clusters to publish for the domain.
17. A computer program product for optimized web domains classification based on progressive crawling with clustering, the computer program product being embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for: crawling a domain to collect data for a subset of pages of a corpus of content associated with the domain; classifying each of the crawled pages into one or more category clusters, wherein the category clusters represent a content categorization of the corpus of content associated with the domain, and wherein the classifying of the each of the crawled pages into the one or more category clusters comprises: determining a category for the each of the crawled pages in the domain; grouping more than one page having the same category into a first cluster; determining whether a number of the more than one page of the first cluster exceeds a first threshold; and in the event that the number of the more than one page of the first cluster does not exceed the first threshold, selecting a new page within the domain to crawl and classify; and determining which of the one or more category clusters to publish for the domain. 18. The computer program product recited in claim 17 , further comprising computer instructions for: determining a sub-entry point to randomly select as a next web page to crawl of the subset of web pages.