title: How to get compressed (text) output from a streaming Hadoop job
body: <p>I have read and tried every example I could find for what seems like this straight forward problem.  Assume there is a set of uncompressed text files and that I want to run a processing step on them and then output a set of compressed files with the results.  To keep things simple, this example assumes <code>cat</code> as the processing step.</p>
  
  <p>Everything I found suggests this should work:  </p>
  
  <pre><code>hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \
      -Dmap.output.compress=true \
      -Dmap.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec \
      -mapper /bin/cat \
      -reducer NONE \
      -input /path_to_uncompressed \
      -output /path_to_compressed
  </code></pre>
  
  <p>The job runs normally, but outputs plain text files.  I have tried varying the input file sizes, varying the codec (Snappy, BZip2, etc.), adding a reducer, setting mapred.output.compression.type (BLOCK, RECROD), etc. and the result is always the same.  For reference, I am using a new install of CDH 4.1.2.</p>