title: how to start sort and reduce in hadoop before shuffle completes for all mappers?
body: <p>I understand from <a href="https://stackoverflow.com/questions/11672676/when-do-reduce-tasks-start-in-hadoop">When do reduce tasks start in Hadoop</a> that the reduce task in hadoop contains three steps: shuffle, sort and reduce where the sort (and after that the reduce) can only start once all the mappers are done. Is there a way to start the sort and reduce every time a mapper finishes.</p>
  
  <p>For example lets we have only one job with mappers mapperA and mapperB and 2 reducers. What i want to do is:</p>
  
  <ol>
  <li>mapperA finishes</li>
  <li>shuffles copies the appropriate partitions of the mapperAs output lets say to reducer 1 and 2</li>
  <li>sort on reducer 1 and 2 starts sorting and reducing and generates some intermediate output</li>
  <li>now mapperB finishes </li>
  <li>shuffles copies the appropriate partitions of the mapperBs output to reducer 1 and 2</li>
  <li>sort and reduce on reducer 1 and 2 starts again and the reducer merges the new output with the old one</li>
  </ol>
  
  <p>Is this possible? Thanks</p>