How can I crawl only a particular domain using Python?
Question: I need to start with a domain and extract all links from the site, then
continue on and extract the links from those links. However, I am only
supposed to traverse the links that are part of the given domain. I'm using
BeautifulSoup. The only way I can think to do this is to test whether the href
is in the form href='www.someotherdomain.com' or of the form href =
'page1/page2'. The former would indicate that the link points to a new domain
and the latter would indicate that the link points to a page within the same
domain.

So I've created a function called has_domain to check to see if the link has
its own domain already. The code looks like this:

    
    
    def has_domain(url):
        if 'www.' in url:
            return True
        else:
            return False
    

The problem is, not every link outside the seed domain starts with 'www.' Is
there an easier way to check to see if a link points to a new domain?


Answer: You can use urlparse to grab the hostname and then check if it's in a list or
compare it directly to a string.

Here's an example of comparing it to a list of domains, notice it treats the
"www" variation as a different domain name.

    
    
    from urlparse import urlparse
    
    url_list = ["example.com", "www.example.com"]
    
    def has_domain(url):
        p = urlparse(url)
        if p.hostname in url_list:
            return True
        else:
            return False
    
    print has_domain("http://www.example.com")
    

