title: How to make Apache Nutch indexing while crawling
body: <p>I started using Apache Nutch (v1.5.1) to index all the website under some certain domain.
  There is huge number of websites (in the order of milions) in my domains and I need to index them step by step instead of waiting the end of the whole process.</p>
  
  <p>I found this in nutch wiki (here <a href="http://wiki.apache.org/nutch/NutchTutorial/#A3.2_Using_Individual_Commands_for_Whole-Web_Crawling" rel="nofollow">http://wiki.apache.org/nutch/NutchTutorial/#A3.2_Using_Individual_Commands_for_Whole-Web_Crawling</a>) something that should work. The idea is to make a script witch calls every single step of my process (crawl, fetch, parse, ...) on a certain amount of data (for example 1000 URL) cyclically.</p>
  
  <pre><code>bin/nutch inject crawl/crawldb crawl/seed.txt
  
  bin/nutch generate crawl/crawldb crawl/segments -topN 25
  s1=`ls -d crawl/segments/2* | tail -1`
  echo $s1
  
  bin/nutch fetch $s1
  bin/nutch parse $s1
  bin/nutch updatedb crawl/crawldb $s1
  
  bin/nutch generate crawl/crawldb crawl/segments -topN 25
  s2=`ls -d crawl/segments/2* | tail -1`
  echo $s2
  
  bin/nutch fetch $s2
  bin/nutch parse $s2
  bin/nutch updatedb crawl/crawldb $s2
  
  ...
  
  bin/nutch invertlinks crawl/linkdb -dir crawl/segments
  bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
  </code></pre>
  
  <p>My question is: is there any way to specify this setting directly into Nutch and make him do this stuff in a parallel and more trasparent way? For example on separated threds?</p>
  
  <p>Thank for answering.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I tried to create the script (the code is above) but unfortunatlly I get an error on the invert link phases. This is the output:</p>
  
  <pre><code>LinkDb: starting at 2012-07-30 11:04:58
  LinkDb: linkdb: crawl/linkdb
  LinkDb: URL normalize: true
  LinkDb: URL filter: true
  LinkDb: internal links will be ignored.
  LinkDb: adding segment: file:/home/apache-nutch-1.5-bin/crawl/segments/20120730102927
  LinkDb: adding segment: file:/home/apache-nutch-1.5-bin/crawl/segments/20120704094625
  ...
  LinkDb: adding segment: file:/home/apache-nutch-1.5-bin/crawl/segments/20120704095730
  
  LinkDb: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
  file:/home/apache-nutch-1.5-bin/crawl/segments/20120730102927/parse_data
  
  Input path does not exist:
  file:/home/apache-nutch-1.5-bin/crawl/segments/20120704094625/parse_data
  ...
  </code></pre>
  
  <p>Thanks for your help.</p>