title: MapReduce: Map-only or Reduce-only?
body: <p>In my problem I have 100TB of data to be processed.  Every file in this dataset is about 1MB, and can belong in up to 3 of the over 10,000 different "groups" we've defined.  Every group of files needs to be processed together, and there can be anywhere from a few to a few hundred files in a group.  Since we have tens of thousands of such groups, we think this is a good candidate for MapReduce.</p>
  
  <p>I see two possible ways to setup this job (perhaps there are more) with something like Hadoop:</p>
  
  <ol>
  <li><p><strong>Map-only</strong>: We archive the files by group, so the splitting and subsequent mapping is done at the group level.  Since every map job has the entire group, it can do the processing itself, and we have no need for a reduce job.  But I see a couple of problems with this solution.  First, since files can exist in up to 3 groups, archiving by group could result in a tripling of our storage overhead, in addition to Hadoop's replication factor.  Furthermore, archiving the data like this would make it less usable in other applications that work with the files differently.</p></li>
  <li><p><strong>Reduce-only</strong>: As I understand it, this paradigm implies a simple "identity" mapper and a data-intensive reducer.  In this solution, the files would be stored unordered on disk, and the mapper would receive a set of files to process.  The mapper would then read the file into memory (at least its header info) to determine what groups it belongs to, then emit (group, file) pairs to be reduced.  The reducer would then be responsible for processing the groups.  However, I worry that we may be losing the benefits of data locality, or bog down the network with too much data traffic, by going this route.</p></li>
  </ol>
  
  <p>Are both methods valid?  If so, which would be preferred?  Specifically, I feel I understand the pros and cons of the Map-only solution fairly well, but not the Reduce-only.  I am not sure how "data local" reduce jobs are, or if it's considered bad practice to do the "heavy-lifting" in the reduce task.</p>