title: MapReduce (secondary) sorting / filtering - how?
body: <p>I have a logfile of timestamped values (concurrent users) of different "zones" of a chatroom webapp in the format "Timestamp; Zone; Value". For each zone exists one value per minute of each day.</p>
  
  <p>For each zone, I want to list the maximum value per day, ordered desc by this maximum value  </p>
  
  <p>So, an input file of</p>
  
  <pre><code>#timestamp; zone; value
  2011-01-01 00:00:00; 1; 10
  2011-01-01 00:00:00; 2; 22
  2011-01-01 00:01:00; 1; 11
  2011-01-01 00:01:00; 2; 21
  
  2011-01-02 00:00:00; 1; 12
  2011-01-02 00:00:00; 2; 20
  </code></pre>
  
  <p>should produce for zone 1:</p>
  
  <pre><code>2011-01-02    12
  2011-01-01    11
  </code></pre>
  
  <p>and for zone 2:</p>
  
  <pre><code>2011-01-01    22
  2011-01-02    20
  </code></pre>
  
  <p>How would I approach this? IMHO I will need more than one M/R step.</p>
  
  <p>What i have implemented so far is:</p>
  
  <ul>
  <li>A mapper that collects a Text-key "YYYY-MM-DD/Zone" and a IntWritable value "value", and</li>
  <li>A reducer that identifies the maximum value per key (i.e. per zone per day).</li>
  </ul>
  
  <p>This results in a file like</p>
  
  <pre><code>2011-01-01/1    11
  2011-01-01/2    22
  2011-01-02/1    12
  2011-01-02/2    20
  </code></pre>
  
  <p>Would this be the input for a second M/R step? If so, what would I take as key and value?</p>
  
  <p>I have studied the "Secondary Sort" example in "Hadoop - The Definitive Guide", but I'm not sure whether and how to apply this here.</p>
  
  <p>Is it possible to M/R into several output-files (one per zone)?</p>
  
  <p><strong>UPDATE</strong>
  After thinking about it, I will try this:</p>
  
  <ul>
  <li>make the key a composite of zone-id and value (using an IntPair?)</li>
  <li>writing a custom KeyComparator and GroupComparator</li>
  </ul>