Python parsing html for complete links urls
Question: I have to parse a html page looking for links in it. Unfortunately, the links
don't contain the full url (for instance starting with
"<http://www.example.com/aResource.html>"). So my parsing get only the
relative URL, for get the whole url address i'm using

    
    
    urlparse.urljoin()
    

But often it leads to some connection errors, and generally i would prefer a
direct way to extract the comlplete urls link. Here is my code:

    
    
    import urlparse
    import requests
    from lxml import html
    from lxml import etree
    
    aFile = requests.get(url)
    tree = html.fromstring(aFile.text)
    
    linkList = tree.xpath('//a')
    
    urls = []
    
    for link in linkList:
        urls.append(str(urlparse.urljoin(url,link.get('href'))))
    

As you can see i'm working with lxml, but i've also tried with BeautifulSoup
without success.


Answer: Since the information (URL scheme, host server, port, path - base URL) is
missing in `<a href=""...>`, it needs to be added to the relative URL.

Usually it is correct to use `urlparse.urljoin()` as you are already doing.

HTML does allow specification of a base url for the page using the `<base
href="...">` tag, which must be defined once in `<head>`. If this tag is
present you should use it's href attribute as your base URL for `urljoin()`.
Your code could be revised to this:

    
    
    import urlparse
    import requests
    from lxml import html
    from lxml import etree
    
    aFile = requests.get(url)
    tree = html.fromstring(aFile.text)
    
    linkList = tree.xpath('//a')
    
    urls = []
    
    try:
        base_url = tree.xpath('//base[1]/@href')[0]
    except IndexError:
        base_url = url
    
    for link in linkList:
        urls.append(str(urlparse.urljoin(base_url,link.get('href'))))
    

However, if you are getting connection errors, it would appear that some of
the links are invalid. Either the base URL derived from the page's URL, or
from the `<base href="...">` tag, is correct. Any invalid URLs constructed
with this value must be due to an invalid relative URL (or invalid `<base>`
tag).

Have you concrete examples of the URL used when connection errors are
experienced?

You could also look at
[`mechanize`](http://wwwsearch.sourceforge.net/mechanize/):

    
    
    import mechanize
    
    br = mechanize.Browser()
    resp = br.open(url)
    urls = [link.absolute_url for link in br.links()]
    

