title: Language recommendations for an efficient web crawler
body: <p>I'm looking for a language for writing an efficient web crawler.  Things I value:</p>
  
  <ul>
  <li>expressive language (don't make me just through static typing hoops)</li>
  <li>useful libraries (a css selector based html parser would be nice)</li>
  <li>minimal memory footprint</li>
  <li>dependable language runtime &amp; libraries</li>
  </ul>
  
  <p>I tried node.js.  I like node in theory.  Javascript is very expressive.  You can use jQuery to parse html.  Node's async nature lets me crawl many urls in parallel without dealing with threads.  V8 is nice and fast for parsing.</p>
  
  <p>In practice, node isn't working out for me.  My process constantly crashes.  Bus Errors, exceptions in the event manager ... etc.</p>
  
  <p>I've done a fair bit of Ruby dev, so I wouldn't mind using Ruby 1.9's coroutines (fibers?) as long as I won't face similar issues with VM / library stability.</p>
  
  <p>Additional suggestions?</p>