title: Fast internet crawler
body: <p>I'd like to do perform data mining on a large scale. For this, I need a fast crawler. All I need is something to download a web page, extract links and follow them recursively, but without visiting the same url twice. Basically, I want to avoid looping.</p>
  
  <p>I already wrote a crawler in python, but it's too slow. I'm not able to saturate a 100Mbit line with it. Top speed is ~40 urls/sec. and for some reason it's hard to get better results. It seems like a problem with python's multithreading/sockets. I also ran into problems with python's gargabe collector, but that was solvable. CPU isn't the bottleneck btw.</p>
  
  <p>So, what should I use to write a crawler that is as fast as possible, and what's the best solution to avoid looping while crawling?</p>
  
  <p>EDIT:
  The solution was to combine <code>multiprocessing</code> and <code>threading</code> modules. Spawn multiple processes with multiple threads per process for best effect. Spawning multiple threads in a single process is not effective and multiple processes with just one thread consume too much memory.</p>