title: Hadoop suitability for recursive data processing
body: <p>I have a filtering algorithm that needs to be applied recursively and I am not sure if MapReduce is suitable for this job. W/o giving too much away, I can say that each object that is being filtered is characterized by a collection if ordered list or queue. </p>
  
  <ol>
  <li>The data is not huge, just about 250MB when I export from SQL to
  CSV. </li>
  <li>The mapping step is simple: the head of the list contains an object that can classify the list as belonging to one of <strong>N</strong> mapping nodes. the filtration algorithm at each node works on the collection of lists assigned to the node and at the end of the filtration, either a list remains the same as before the filtration or the head of the list is removed.</li>
  <li>The reduce function is simple too: all the map jobs' lists are brought together and may have to be written back to disk.</li>
  <li>When all the <strong>N</strong> nodes have returned their output, the mapping step is repeated with this new set of data.</li>
  </ol>
  
  <p>Note: <strong>N</strong> can be as much as 2000 nodes.
  Simple, but it requires perhaps up to a 1000 recursions before the algorithm's termination conditions are met.</p>
  
  <p>My question is would this job be suitable for Hadoop? If not, what are my options?</p>