title: Optimizing python script with multithreading
body: <p>Hello Guys! I have written small web page crawler functions. But I am new to multithreading, I could not optimize it. My code is:</p>
  
  <pre><code>alreadySeenURLs = dict() #the dictionary of already seen crawlers
  candidates = set() #the set of URL candidates to crawl
  
  def initializeCandidates(url):
  
      #gets page with urllib2
      page = getPage(url)
  
      #parses page with BeautifulSoup
      parsedPage = getParsedPage(page)
  
      #function which return all links from parsed page as set
      initialURLsFromRoot = getLinksFromParsedPage(parsedPage)
  
      return initialURLsFromRoot 
  
  def updateCandidates(oldCandidates, newCandidates):
      return oldCandidates.union(newCandidates)
  
  candidates = initializeCandidates(rootURL)
  
  for url in candidates:
  
      print len(candidates)
  
      #fingerprint of URL
      fp = hashlib.sha1(url).hexdigest()
  
      #checking whether url is in alreadySeenURLs
      if fp in alreadySeenURLs:
          continue
  
      alreadySeenURLs[fp] = url
  
      #do some processing
      print url
  
      page = getPage(url)
      parsedPage = getParsedPage(page, fix=True)
      newCandidates = getLinksFromParsedPage(parsedPage)
  
      candidates = updateCandidates(candidates, newCandidates)
  </code></pre>
  
  <p>As one can see, here it takes one url from candidates at specific time. I wanted to make this script multithreaded, in such a way, that it could take at least N urls from candidates, and do the job. Can anyone guide me? Give any links or suggestions? </p>