24. A method for configuring a web crawl, the method comprising the steps of: specifying a starter seed uniform resource locator (URL) and a user-specified web crawl configuration to a configurable web crawler which crawls a web to inspect text of page resources accessible on the web, each page resource identified by a unique URL, the user-specified web crawl configuration comprising user-specified crawling rules specifying crawling behavior, the crawling rules specifying crawling behavior for at least one each of a general crawl behavior, one or more page-specific crawl behavior to be applied during a crawl to one or more identified pages, and one or more element-specific crawl behavior to be applied during a crawl to one or more identified page elements, wherein the received crawling rules define the crawling behavior in terms of whether to record the resource text, whether to follow hyperlinks of the resource, whether to record text description in metadata associated with the page, whether to record keywords identified in the metadata associated with the page, whether to override a page title element, whether to exclude one or more identified resource pages, whether to exclude one or more identified domains; wherein the configurable web crawler is configured to maintain a queue of unprocessed URLs of resources, insert the specified starter seed URL into the queue of unprocessed URLs of resources, retrieve from the queue a next unprocessed URL, retrieve the user-specified web crawl configuration which specifies the crawling rules, and if allowed by the specified general crawl behavior and page-specific crawl behavior of the retrieved crawling rules, fetch the page resource addressed by the retrieved URL and process the fetched page resource on an element-by-element basis by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior; and repeating the processing until either all elements have been parsed and processed in accordance with the retrieved crawling rules, and wherein the default crawl behavior comprises extracting URLs of outbound links and adding the extracted URLs to the queue, storing text to a resource repository for further analysis, the one or more processing units repeating the processing on available unprocessed URLs in the queue.
24. A method for configuring a web crawl, the method comprising the steps of: specifying a starter seed uniform resource locator (URL) and a user-specified web crawl configuration to a configurable web crawler which crawls a web to inspect text of page resources accessible on the web, each page resource identified by a unique URL, the user-specified web crawl configuration comprising user-specified crawling rules specifying crawling behavior, the crawling rules specifying crawling behavior for at least one each of a general crawl behavior, one or more page-specific crawl behavior to be applied during a crawl to one or more identified pages, and one or more element-specific crawl behavior to be applied during a crawl to one or more identified page elements, wherein the received crawling rules define the crawling behavior in terms of whether to record the resource text, whether to follow hyperlinks of the resource, whether to record text description in metadata associated with the page, whether to record keywords identified in the metadata associated with the page, whether to override a page title element, whether to exclude one or more identified resource pages, whether to exclude one or more identified domains; wherein the configurable web crawler is configured to maintain a queue of unprocessed URLs of resources, insert the specified starter seed URL into the queue of unprocessed URLs of resources, retrieve from the queue a next unprocessed URL, retrieve the user-specified web crawl configuration which specifies the crawling rules, and if allowed by the specified general crawl behavior and page-specific crawl behavior of the retrieved crawling rules, fetch the page resource addressed by the retrieved URL and process the fetched page resource on an element-by-element basis by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior; and repeating the processing until either all elements have been parsed and processed in accordance with the retrieved crawling rules, and wherein the default crawl behavior comprises extracting URLs of outbound links and adding the extracted URLs to the queue, storing text to a resource repository for further analysis, the one or more processing units repeating the processing on available unprocessed URLs in the queue. 30. The method of claim 24 , wherein the crawling rules further specify crawling behavior for one or more domain-specific crawl behavior to be applied during a crawl to one or more identified domains; wherein the web crawler is configured to, only if allowed by the specified general crawl behavior, the domain-specific crawl behavior, and page-specific crawl behavior of the retrieved crawling rules, fetch the page resource addressed by the retrieved URL and to process the fetched page resource on an element-by-element basis by parsing a next page element from the fetched page resource, and if an element-specific behavior is specified in the retrieved crawling rules for the parsed next page element then operating with regard to the element according to the specified element-specific behavior, and otherwise if a page-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified page-specific behavior, and otherwise if a domain-specific behavior is specified for the fetched page resource then operating with regard to the element according to the specified domain-specific behavior, and otherwise if a global crawl behavior is specified then operating with regard to the element according to the specified global crawl behavior, and otherwise operating with regard to the element according to a default crawl behavior; and repeating the processing until either all elements have been parsed and processed in accordance with the retrieved crawling rules.