title: Using re.findall() in Python for Web Crawling
body: <p>I am trying to teach myself Python by writing a very simple web crawler with it.</p>
  
  <p>The code for it is here:</p>
  
  <pre><code>#!/usr/bin/python
  
  import sys, getopt, time, urllib, re
  
  LINK_INDEX = 1
  links = [sys.argv[len(sys.argv) - 1]]
  visited = []
  politeness = 10
  maxpages = 20
  
  def print_usage():
      print "USAGE:\n./crawl [-politeness &lt;seconds&gt;] [-maxpages &lt;pages&gt;] seed_url"
  
  def parse_args():
      #code for parsing arguments (works fine so didnt need to be included here)
  
  def crawl():
      global links, visited
      url = links.pop()    
      visited.append(url)
  
      print "\ncurrent url: %s" % url
  
      response = urllib.urlopen(url)
      html = response.read()
  
      html = html.lower()
  
      raw_links = re.findall(r'&lt;a href="[\w\.-]+"', html)
  
      print "found: %d" % len(raw_links)
  
      for raw_link in raw_links:
          temp = raw_link.split('"')
          if temp[LINK_INDEX] not in visited and temp[LINK_INDEX] not in links:
              links.append(temp[LINK_INDEX])
  
      print "\nunvisited:"
      for link in links:
          print link
  
      print "\nvisited:"
      for link in visited:
          print link
  
  parse_args()
  
  while len(visited) &lt; maxpages and len(links) &gt; 0:
      crawl()
      time.sleep(politeness)
  
  print "politeness = %d, maxpages = %d" % (politeness, maxpages)
  </code></pre>
  
  <p>I created a small test network in the same working directory of about 10 pages that all link together in various ways, and it seems to work fine, but when I send it out onto the actual internet by itself, it is unable to parse links from files it gets.</p>
  
  <p>It is able to get the html code fine, because I can print that out, but it seems that the <code>re.findall()</code> part is not doing what it is supposed to, because the links list never gets populated. Have I maybe written my regex wrong? It worked fine to find strings like <code>&lt;a href="test02.html"</code> and then parse the link from that, but for some reason, it isn't working for actual web pages. It might be the http part perhaps that is throwing it off?</p>
  
  <p>I've never used regex with Python before so I'm pretty sure that this is the problem. Can anyone give me any idea how express the pattern I am looking for better? Thanks!</p>