fast parsing links out of a page in python
Question: I need to parse a large number of pages (say 1000) and replace the links with
tinyurl links.

right now i am doing this using a regex

    
    
    href_link_re = re.compile(r"<a[^>]+?href\s*=\s*(\"|')(.*?)\1[^>]*>", re.S)
    

but its not fast enough.

i am thinking so far

  1. state machine (the success of this will depend on my ability to write clever code)
  2. using an html parser 

Can you suggest **faster** ways?

**EDIT:** You would think that an html parser would be faster than regex, but
in my tests it is not:

    
    
    from BeautifulSoup import BeautifulSoup, SoupStrainer
    
    import re
    import time
    
    __author__ = 'misha'
    
    regex = re.compile(r"<a[^>]+?href\s*=\s*(\"|')(.*?)\1[^>]*>", re.S)
    
    def test(text, fn, desc):
    
        start = time.time()
        total  = 0
        links = [];
        for i in range(0, 10):
            links = fn(text)
            total += len(links)
        end = time.time()
        print(desc % (end-start, total))
       # print(links)
    
    def parseRegex(text):
        links  = set([])
        for link in regex.findall(text):
            links.add(link[1])
        return links
    
    def parseSoup(text):
        links = set([])
        for link in BeautifulSoup(text, parseOnlyThese=SoupStrainer('a')):
            if link.has_key('href'):
                links.add(link['href'])
    
        return links
    
    
    
    if __name__ == '__main__':
        f = open('/Users/misha/test')
        text = ''.join(f.readlines())
        f.close()
    
        test(text, parseRegex, "regex time taken: %s found links: %s" )
        test(text, parseSoup, "soup time taken: %s found links: %s" )
    

output:

    
    
    regex time taken: 0.00451803207397 found links: 2450
    soup time taken: 0.791836977005 found links: 2450
    

(test is a dump of the wikipedia front page)

i must be using soup badly. **what am i doing wrong?**


Answer: LXML is probably your best bet for this task. See [Beautiful Soup vs LXML
Performance](http://blog.dispatched.ch/2010/08/16/beautifulsoup-vs-lxml-
performance/). Parsing links is easy in LXML and it's fast.

    
    
    root = lxml.html.fromstring(s)
    anchors = root.cssselect("a")
    links = [a.get("href") for a in anchors]
    

