10. The method of claim 1 , further comprising adapting, by the application, one or more of the plurality of secondary web site categorization rules based on the counting, for each category of web page, of the number of occurrences of different keywords in the URLs stored in the pool of validated URLs that are associated to the category.
1. A method of crawling web pages and categorizing the web pages in a two-pass process used for delivering targeted content to a user equipment (UE) based on categories of web sites browsed by the UE, comprising: for each uniform resource locater (URL) stored in a pool of unvalidated URLs; removing a URL from the pool of unvalidated URLs by an application executing on a server computer; navigating to a web page referenced by the URL by the application, if navigating to the URL results in a redirect; discarding the URL, removed from the pool of unvalidated URLs by the application; searching for keywords in the URL and in a title of the web page referenced by the URL by the application; searching for URLs embedded in the web page referenced by the URL by the application; adding discovered embedded URLs to the pool of unvalidated URLs by the application; executing a plurality of primary web site categorization rules on the keywords associated with the web page referenced by the URL by the application, where each category primary rule is associated with a single category of web page, different category primary rules associate to different categories of web pages, and each category primary rue comprises a logical statement that is true if the keywords discovered in the title of the web page referenced by the URL or in the URL make the logical statement true; associating the URL to each category the category primary rule evaluates true by the application; executing a plurality of secondary web site categorization rules on the keywords associated with the web page referenced by the URL by the application, where each category secondary rule is associated with a single category of web page, different category secondary rules associate to different categories of web pages, and each category secondary rule comprises a logical statement that is true if the keywords discovered in the title of the web page referenced by the URL or in the URL make the logical statement true; associating the URL to each category the category secondary rule evaluates true by the application; and saving the URL the keywords associated with the web page referenced by the URL, and identities of categories to which the URL is associated in a pool of validated URLs by the application; for each category of web page; counting the number of occurrences of different keywords in the URLs stored in the pool of validated URLs that are associated to the category by the application; counting the number of URLs stored in the pool of validated URLs that are associated to the category by the application; for each different keyword that is not identified in a category primary rule or in a category secondary rule, determining by the application a percentage of the URLs that are associated to the category that are associated to the different keyword not identified in a rule versus the number of URLs stored in the pool of validated URLs that are associated to the category and for each said percentage that exceeds a predefined threshold percentage extending the logic of the category secondary rule by including the keyword associated with that said percentage; analyzing a web browsing history of a UE to determine URLs accessed by the UE that ace stored in the pool of validated URLs; associating the to one or more categories associated to the browsed URLs that are stored in the pool of validated URLs; and sending content that is targeted to the UE based on the one or more categories associated to the UE.