title: fast parsing links out of a page in python
body: <p>I need to parse a large number of pages (say 1000) and replace the links with tinyurl links.</p>
  
  <p>right now i am doing this using a regex </p>
  
  <pre><code>href_link_re = re.compile(r"&lt;a[^&gt;]+?href\s*=\s*(\"|')(.*?)\1[^&gt;]*&gt;", re.S)
  </code></pre>
  
  <p>but its not fast enough. </p>
  
  <p>i am thinking so far</p>
  
  <ol>
  <li>state machine (the success of this will depend on my ability to write clever code)</li>
  <li>using an html parser </li>
  </ol>
  
  <p>Can you suggest <strong>faster</strong> ways?</p>
  
  <p><strong>EDIT:</strong>
  You would think that an html parser would be faster than regex, but in my tests it is not:</p>
  
  <pre><code>from BeautifulSoup import BeautifulSoup, SoupStrainer
  
  import re
  import time
  
  __author__ = 'misha'
  
  regex = re.compile(r"&lt;a[^&gt;]+?href\s*=\s*(\"|')(.*?)\1[^&gt;]*&gt;", re.S)
  
  def test(text, fn, desc):
  
      start = time.time()
      total  = 0
      links = [];
      for i in range(0, 10):
          links = fn(text)
          total += len(links)
      end = time.time()
      print(desc % (end-start, total))
     # print(links)
  
  def parseRegex(text):
      links  = set([])
      for link in regex.findall(text):
          links.add(link[1])
      return links
  
  def parseSoup(text):
      links = set([])
      for link in BeautifulSoup(text, parseOnlyThese=SoupStrainer('a')):
          if link.has_key('href'):
              links.add(link['href'])
  
      return links
  
  
  
  if __name__ == '__main__':
      f = open('/Users/misha/test')
      text = ''.join(f.readlines())
      f.close()
  
      test(text, parseRegex, "regex time taken: %s found links: %s" )
      test(text, parseSoup, "soup time taken: %s found links: %s" )
  </code></pre>
  
  <p>output:</p>
  
  <pre><code>regex time taken: 0.00451803207397 found links: 2450
  soup time taken: 0.791836977005 found links: 2450
  </code></pre>
  
  <p>(test is a dump of the wikipedia front page)</p>
  
  <p>i must be using soup badly. <strong>what am i doing wrong?</strong></p>