title: Hadoop secondary sort for values. Sorting, gorping loose values
body: <p>It's rather common problem, I don't understand what to pick.</p>
  
  <p>I have fields: <strong>id, creationDate, state, dateDiff</strong></p>
  
  <p><strong>id</strong> is a <strong>natural key</strong>.</p>
  
  <p>I need to get in my reducer:</p>
  
  <p><strong>KEY(id), VALUE(creationDate, state, dateDiff)</strong></p>
  
  <p><strong>VALUE(creationDate, state, dateDiff) should be sorted by: creationDate, state</strong></p>
  
  <p>What key should I have to pick?
  I did create <strong>composite key (id, creationDate, state)</strong></p>
  
  <p>I did <strong>implement</strong> 
  partitioner by <strong>id</strong></p>
  
  <p>grouper by <strong>id</strong></p>
  
  <p><strong>sorter</strong> by <strong>id, creationDate,state</strong></p>
  
  <p>My reducer gets only unique id...
  For example:</p>
  
  <pre><code>1 123 true  6
  1 456 false 6
  1 789 true  7
  </code></pre>
  
  <p>I get only</p>
  
  <pre><code>1 123 true  6
  </code></pre>
  
  <p>In my reducer. Seems like I don't get sorter, partitioner, grouper :( There is a luxk of understanding.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>public class POIMapper extends Mapper&lt;LongWritable, Text, XVLRKey, XVLRValue&gt;{
  
      private static final Log LOG = LogFactory.getLog(POIMapper.class);
  
      @Override
      public void map(LongWritable key, Text csvLine, Context context) throws IOException, InterruptedException {
          Pair&lt;XVLRKey, XVLRValue&gt; xvlrPair = POIUtil.parseKeyAndValue(csvLine.toString(), POIUtil.CSV_DELIMITER);
          context.write(xvlrPair.getValue0(), xvlrPair.getValue1());
      }
  
  }
  
  public class POIReducer extends Reducer&lt;XVLRKey, XVLRValue, LongWritable, Text&gt;{
  
      private static final Log LOG = LogFactory.getLog(POIReducer.class);
  
      private final Text textForOutput = new Text();
  
      @Override()
      public void reduce(XVLRKey key, Iterable&lt;XVLRValue&gt; values, Context context)
                                                                              throws IOException, InterruptedException {
          XVLROutput out = null;
  //Just check that values are correctly attached to keys. No logic here...
          LOG.info("\nPOIReducer: key:"+key);
          for(XVLRValue value : values){
              LOG.info("\n --- --- --- value:"+value+"\n");
              textForOutput.set(print(key, value));
              context.write(key.getMsisdn(), textForOutput);
          }
      }
  
      private String print(XVLRKey key, XVLRValue value){
          StringBuilder builder = new StringBuilder();
          builder.append(value.getLac())          .append("\t")
                 .append(value.getCellId())       .append("\t")
                 .append(key.getDateOccurrence()) .append("\t")
                 .append(value.getTimeDelta());
          return builder.toString();
      }
  }
  </code></pre>
  
  <p>Job code:</p>
  
  <pre><code>JobBuilder&lt;POIJob&gt; jobBuilder = createTestableJobInstance();
  
          jobBuilder.withOutputKey(XVLRKey.class);
          jobBuilder.withOutputValue(XVLRValue.class);
  
          jobBuilder.withMapper(POIMapper.class);
          jobBuilder.withReducer(POIReducer.class);
  
          jobBuilder.withInputFormat(TextInputFormat.class);
          jobBuilder.withOutputFormat(TextOutputFormat.class);
  
          jobBuilder.withPartitioner(XVLRKeyPartitioner.class);
          jobBuilder.withSortComparator(XVLRCompositeKeyComparator.class);
          jobBuilder.withGroupingComparator(XVLRKeyGroupingComparator.class);
  
          boolean result = buildSubmitAndWaitForCompletion(jobBuilder);
          MatcherAssert.assertThat(result, Matchers.is(true));
  
  
  
  
  public class XVLRKeyPartitioner extends Partitioner&lt;XVLRKey, XVLRValue&gt; {
  
      @Override
      public int getPartition(XVLRKey key, XVLRValue value, int numPartitions) {
              return Math.abs(key.getMsisdn().hashCode() * 127) % numPartitions;
      }
  }
  
  public class XVLRCompositeKeyComparator extends WritableComparator {
  
      protected XVLRCompositeKeyComparator() {
          super(XVLRKey.class, true);
      }
  
      @Override
      public int compare(WritableComparable writable1, WritableComparable writable2) {
          XVLRKey key1 = (XVLRKey) writable1;
          XVLRKey key2 = (XVLRKey) writable2;
         return key1.compareTo(key2);
      }
  }
  
  public class XVLRKeyGroupingComparator extends WritableComparator {
  
      protected XVLRKeyGroupingComparator() {
          super(XVLRKey.class, true);
      }
  
      @Override
      public int compare(WritableComparable writable1, WritableComparable writable2) {
  
          XVLRKey key1 = (XVLRKey) writable1;
          XVLRKey key2 = (XVLRKey) writable2;
  
          return key1.getMsisdn().compareTo(key2.getMsisdn());
  
      }
  }
  
  public class XVLRKey implements WritableComparable&lt;XVLRKey&gt;{
  
      private  final LongWritable msisdn;
      private  final LongWritable dateOccurrence;
      private  final BooleanWritable state;
  //getters-setters
  }
  
  public class XVLRValue implements WritableComparable&lt;XVLRValue&gt; {
  
      private final LongWritable lac;
      private final LongWritable cellId;
      private final LongWritable timeDelta;
      private final LongWritable dateOccurrence;
      private final BooleanWritable state;
  //getters-setterrs
  }
  </code></pre>
  
  <p>Please see that XVLRKey, XVLRValue do have duplicated fields. I've duplicated dateOccurrence, state in XVLRKey because I want to get sorted values in my reducer. They should be sorted by dateOccurrence.</p>
  
  <p>I can't find a way how to solve this problem without duplication.</p>