title: Hadoop mapreduce defining separators for streaming
body: <p>I'm using <code>Hadoop 2.7.1</code> </p>
  
  <p>I'm really struggling to understand at what point in the <strong>streaming</strong> process sorts are applied, how you can change the sort order, and the separator. Reading the <a href="https://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html" rel="nofollow">documentation</a> has confused me further since some config variables seem to be used interchangeably (c.f. <code>map.output.key.field.separator</code> used in the <a href="https://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html#Hadoop_Partitioner_Class" rel="nofollow">partitioner example</a> and <code>mapreduce.map.output.key.field.separator</code> in the <a href="https://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html#Hadoop_Comparator_Class" rel="nofollow">comparator example</a>.</p>
  
  <h2>My questions</h2>
  
  <ol>
  <li>At what point in the process are each of these configuration variables used and what exactly do they affect (and <em>is there a way by looking at them I can tell?</em>)</li>
  <li><p>Additionally, what happens when they are used together? Do they over-ride each other?</p>
  
  <ul>
  <li><code>mapreduce.map.output.key.field.separator</code></li>
  <li><code>map.output.key.field.separator</code></li>
  <li><code>stream.map.output.field.separator</code></li>
  <li><code>stream.num.map.output.key.fields</code></li>
  <li><code>mapreduce.partition.keypartitioner.options</code></li>
  <li><code>mapred.text.key.partitioner.options</code></li>
  <li><code>mapreduce.fieldsel.data.field.separator</code></li>
  <li><code>mapreduce.fieldsel.map.output.key.value.fields.spec</code></li>
  <li><code>mapreduce.fieldsel.reduce.output.key.value.fields.spec</code></li>
  <li><code>stream.reduce.input.field.separator</code></li>
  <li><code>stream.num.reduce.input.fields</code></li>
  <li><code>stream.reduce.output.field.separator</code></li>
  <li><code>stream.num.reduce.output.fields</code></li>
  </ul></li>
  </ol>
  
  <p>Just search the <a href="https://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html" rel="nofollow">streaming documentation</a> to see the context for their introduction. Most are (not contained in <a href="https://hadoop.apache.org/docs/r2.7.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml" rel="nofollow">mapred-default.xml</a>).</p>
  
  <p>I feel my confusion may be as a result of the distinction between separating the records into keys and values, and separating the key into multiple fields.</p>
  
  <p>My understanding of the process (ignoring combiners for now):</p>
  
  <ol>
  <li>Input data splits sent to the <code>Mappers</code></li>
  <li><code>Mapper</code> outputs are split into (<code>key</code>, <code>value</code>) pairs and sent to the <code>Partitioner</code></li>
  <li><code>Partitioner</code> sends the <code>Mapper</code> outputs to <code>Comparator</code> based on the key (<em>and the fields within the key?</em>)</li>
  <li>The <code>Comparator</code> sorts each partition and sends it to a <code>Reducer</code></li>
  </ol>