title: Anybody knows a good extendable open source web-crawler?
body: <p>The crawler needs to have an extendable architecture to allow changing the internal process, like implementing new steps (pre-parser, parser, etc...)</p>
  
  <p>I found the Heritrix Project (<a href="http://crawler.archive.org/" rel="noreferrer">http://crawler.archive.org/</a>). </p>
  
  <p>But there are other nice projects like that?</p>