1. A method of crawling web pages and categorizing the web pages in a two-pass process used for delivering targeted content to a user equipment (UE) based on categories of web sites browsed by the UE, comprising: for each uniform resource locater (URL) stored in a pool of unvalidated URLs; removing a URL from the pool of unvalidated URLs by an application executing on a server computer; navigating to a web page referenced by the URL by the application, if navigating to the URL results in a redirect; discarding the URL, removed from the pool of unvalidated URLs by the application; searching for keywords in the URL and in a title of the web page referenced by the URL by the application; searching for URLs embedded in the web page referenced by the URL by the application; adding discovered embedded URLs to the pool of unvalidated URLs by the application; executing a plurality of primary web site categorization rules on the keywords associated with the web page referenced by the URL by the application, where each category primary rule is associated with a single category of web page, different category primary rules associate to different categories of web pages, and each category primary rue comprises a logical statement that is true if the keywords discovered in the title of the web page referenced by the URL or in the URL make the logical statement true; associating the URL to each category the category primary rule evaluates true by the application; executing a plurality of secondary web site categorization rules on the keywords associated with the web page referenced by the URL by the application, where each category secondary rule is associated with a single category of web page, different category secondary rules associate to different categories of web pages, and each category secondary rule comprises a logical statement that is true if the keywords discovered in the title of the web page referenced by the URL or in the URL make the logical statement true; associating the URL to each category the category secondary rule evaluates true by the application; and saving the URL the keywords associated with the web page referenced by the URL, and identities of categories to which the URL is associated in a pool of validated URLs by the application; for each category of web page; counting the number of occurrences of different keywords in the URLs stored in the pool of validated URLs that are associated to the category by the application; counting the number of URLs stored in the pool of validated URLs that are associated to the category by the application; for each different keyword that is not identified in a category primary rule or in a category secondary rule, determining by the application a percentage of the URLs that are associated to the category that are associated to the different keyword not identified in a rule versus the number of URLs stored in the pool of validated URLs that are associated to the category and for each said percentage that exceeds a predefined threshold percentage extending the logic of the category secondary rule by including the keyword associated with that said percentage; analyzing a web browsing history of a UE to determine URLs accessed by the UE that ace stored in the pool of validated URLs; associating the to one or more categories associated to the browsed URLs that are stored in the pool of validated URLs; and sending content that is targeted to the UE based on the one or more categories associated to the UE.