title: Best open source, extendable crawler to use for image crawling
body: <p>We are in the starting phase of a project, and we are currently wondering
  whether which crawler is the best choice for us.</p>
  
  <p>Our project:</p>
  
  <p>Basically, we're going to set up Hadoop and crawl the web for images.
  We will then run our own indexing software on the images stored in HDFS
  based on the Map/Reduce facility in Hadoop. We will not use other indexing
  than our own.</p>
  
  <p>Some particular questions:</p>
  
  <ul>
  <li>Which crawler will handle crawling for images best?</li>
  <li>Which crawler will best adapt to a distributed crawling system, in which we
  use many servers conducting crawling together?</li>
  </ul>
  
  <p>Right now these look like the 3 best options-</p>
  
  <ul>
  <li>Nutch: Known to scale. Doesn't look like the best option because it seems that is it tied closely to their text searching software.</li>
  <li>Heritrix: Also scales. This one currently looks like the best option.</li>
  <li>Scrapy: Has not been used on a large scale (not sure though). I dont know if it has the basic stuff like URL canonicalization. I would like to use this one because it is a python framework (I like python more than java), but I don't know if they have implemented the advanced features of a web crawler.</li>
  </ul>
  
  <p><strong>Summary:</strong></p>
  
  <p>We need to get as many images as possible from the web. Which existing crawling framework is both scalable and efficient , but also the one which will be the easiest to modify to get only images?</p>
  
  <p>Thanks!</p>