title: Hadoop for processing data from Apache solr
body: <p>I have to process some huge amount of data.I would like it to be processed using distributed computing(Scalable).
  I am fetching data from apache Solr.On passing a particular input i get a huge dataset  from apache solr.For each record in this dataset i will  pass the primary key to a REST api to obtain some information which will be  attached to the record.Then each record will undergo some update.Each updated object in final huge collection will be written as seperate xml files into a folder.</p>
  
  <p>Is hadoop applicable in this particular scenario?.I have seen the wordcount sample in hadoop mapreduce documentation.I tried to think of my situation in a similar way in which map emitted by map reduce for 2 nodes will be</p>
  
  <pre><code>Node1 - Map&lt;InputIdToSolr1,Set&lt;RecordsFromSolr1to500&gt;&gt;
  Node2 - Map&lt;InputIdToSolr1,Set&lt;RecordsFromSolr500to1000&gt;&gt;
  </code></pre>
  
  <p>Then this results will be combined by the reduce function in hadoop.Unlike wordcount my nodes will have only one element in map for each node.I am not sure if using hadoop makes sense.
  What are other options/open source java projects i can use to  scale the processing of records.I have seen Terracotta from spring but it seems to be a commercial application.</p>