title: Creating a Web Crawler in Ruby. How to tackle performance issues?
body: <p>I'm building a web crawler in Ruby, Rails as the front-end. I'm using Mechanize which is built on top of Nokogiri. I already implemented a solution that will crawl the web pages but I looking to be able to crawl 200k websites in a single run and I know there's a better way than  waiting for hours on it to finish. I'm want to be able to achieve the best performance by firing up parallel requests without making it too complex. I don't know anything about threading and what's the limit on it so don't hold the server hostage while the crawler is running if someone would like to point where I can learn how to do this or at least tell me what should I be looking for. Keep in my that I will be writing to the database and to a file (probably I can export form the database once the crawl is done and not write to the file directly). Thanks.</p>
  
  <p>Note: There's a similar question here in SO but is a few years old maybe people are doing it differently now a days and seems very complex.</p>