title: How can I set block size of output files produced by a Hadoop Streaming job?
body: <p>Seems like this should be simple; I have a set of files on our cluster with the cluster-default block size of 128MB.  I have a streaming job that process them, and I would like the output files created by the streaming job to use a different block size, specifically 16MB.  I thought the following would work:</p>
  
  <blockquote>
    <p>$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-0.20.1+152-streaming.jar -D dfs.block.size=16777216 -D mapred.job.name='Log Processor' -D mapred.reduce.tasks=5 -D mapred.output.compress=true -D mapred.output.compression.type=BLOCK -input /production/beacon/vbox*/20110609/00/access_log -output /user/me/logprocess/20110609/00/access_log -mapper /bin/cat -outputformat org.apache.hadoop.mapred.SequenceFileOutputFormat</p>
  </blockquote>
  
  <p>No love, the files produced at /user/me/logprocess/20110609/00/access_log all use the default block size for the cluster of 128MB.  I've tried replacing dfs.block.size with fs.local.block.size in the command as well, with the same results.  Any ideas how to control this in a streaming job?</p>