Find and append each reference to a html link - Python
Question: I have a HTML file I got from Wikipedia and would like to find every link on
the page such as `/wiki/Absinthe` and replace it with the current directory
added to the front such as `/home/fergus/wikiget/wiki/Absinthe` so for:

    
    
    <a href="/wiki/Absinthe">Absinthe</a>
    

becomes:

    
    
    <a href="/home/fergus/wikiget/wiki/Absinthe">Absinthe</a>
    

and this is throughout the whole document.

Do you have any ideas? I'm happy to use BeautifulSoup or Regex!


Answer: If that's really all you have to do, you could do it with `sed` and its `-i`
option to rewrite the file in-place:

    
    
    sed -e 's,href="/wiki,href="/home/fergus/wikiget/wiki,' wiki-file.html
    

However, here's a Python solution using the lovely [lxml](http://lxml.de/)
API, in case you need to do anything more complex or you might have badly
formed HTML, etc.:

    
    
    from lxml import etree
    import re
    
    parser = etree.HTMLParser()
    
    with open("wiki-file.html") as fp:
        tree = etree.parse(fp, parser)
    
    for e in tree.xpath("//a[@href]"):
        link = e.attrib['href']
        if re.search('^/wiki',link):
            e.attrib['href'] = '/home/fergus/wikiget'+link
    
    # Or you can just specify the same filename to overwrite it:
    with open("wiki-file-rewritten.html","w") as fp:
        fp.write(etree.tostring(tree))
    

Note that `lxml` is probably a better option than BeautifulSoup for this kind
of task nowadays, for the
[reasons](http://www.crummy.com/software/BeautifulSoup/3.1-problems.html)
given by BeautifulSoup's author.

