title: How do I store crawled data into a database
body: <p>I'm fairly new to python and everything else I'm about to talk about in this question but I want to get started with a project I've been thinking about for sometime now. Basically I want to crawl the web and display the urls as and when they are crawled in-real time on the web page. I coded a simple crawler which stores the urls in a list. I was wondering how to get this list into a database and have the database updated every x seconds, so that I can access the database and output the list of links on the web page periodically.</p>
  
  <p>I don't know so much about real-time web development but that's a topic for another day. Right now though, I'm more concerned about how to get the list into the database. I'm currently using the web2py framework which is quite easy to get along with but if you guys have any recommendations as to where I should look, what frameworks I should check out... please do comment that too in your answers, thanks.</p>
  
  <p>In a nutshell, the things I'm a noob at are: Python, databases, real-time web dev.</p>
  
  <p>here's the code to my crawler if it helps in anyway :) thanks</p>
  
  <pre><code>from urllib2 import urlopen
  def crawler(url,x):
      crawled=[]
      tocrawl=[]
      def crawl(url,x):
          x=x+1
          try:
              page = urlopen(url).read()
              findlink = page.find('&lt;a href=')
              if findlink == -1:
                  return None, 0
              while findlink!=-1:
                  start = page.find(('"'), findlink)
                  end = page.find(('"'), start+1)
                  link = page[start+1:end]
                  if link:
                      if link!=url:
                          if link[0]=='/':
                              link=url+link
                              link=replace(link)
                          if (link not in tocrawl) and (link!="") and (link not in crawled):
                              tocrawl.append(link)
                  findlink = page.find('&lt;a href=', end)
              crawled.append(url)
              while tocrawl:
                  crawl(tocrawl[x],x)
          except:
              #keep crawling
              crawl(tocrawl[x],x)
      crawl(url,x)
  
  
  def replace(link):
      tsp=link.find('//')
      if tsp==-1:
          return link
      link=link[0:tsp]+'/'+link[tsp+2:]
      return link
  </code></pre>