1. A method comprising: maintaining a URL database comprising a plurality of URLs; retrieving, without user intervention, a first document associated with a first URL from the plurality of URLs; parsing, using at least one processor, the first document to identify one or more additional URLs within the first document; comparing the one or more additional URLs to the URL database to determine if the URL database includes the one or more additional URLs; if it is determined that the URL database does not include the one or more additional URLs, updating the URL database to include the one or more additional URLs; determining if the one or more additional URLs are root URLs based on a set of predefined rules; when it is determined that the one or more additional URLs are root URLs, classifying the one or more additional URLs as root URLs in the URL database and updating the URL database to add the classification that the one or more additional URLs are root URLs; querying a third-party database for additional information associated with the one or more additional URLs; updating the URL database to include the additional information associated with the one or more additional URLs; and repeating the steps of retrieving, parsing, comparing, and updating for each of the one or more additional URLs.
1. A method comprising: maintaining a URL database comprising a plurality of URLs; retrieving, without user intervention, a first document associated with a first URL from the plurality of URLs; parsing, using at least one processor, the first document to identify one or more additional URLs within the first document; comparing the one or more additional URLs to the URL database to determine if the URL database includes the one or more additional URLs; if it is determined that the URL database does not include the one or more additional URLs, updating the URL database to include the one or more additional URLs; determining if the one or more additional URLs are root URLs based on a set of predefined rules; when it is determined that the one or more additional URLs are root URLs, classifying the one or more additional URLs as root URLs in the URL database and updating the URL database to add the classification that the one or more additional URLs are root URLs; querying a third-party database for additional information associated with the one or more additional URLs; updating the URL database to include the additional information associated with the one or more additional URLs; and repeating the steps of retrieving, parsing, comparing, and updating for each of the one or more additional URLs. 7. The method of claim 1 , further comprising, if it is determined that the one or more additional URLs are not root URLs, classifying the one or more additional URLs as leaf URLs in the URL database.