title: How to use Hadoop Streaming with LZO-compressed Sequence Files?
body: <p>I'm trying to play around with the Google ngrams dataset using Amazon's Elastic Map Reduce. There's a public dataset at <a href="http://aws.amazon.com/datasets/8172056142375670" rel="noreferrer">http://aws.amazon.com/datasets/8172056142375670</a>, and I want to use Hadoop streaming.</p>
  
  <p>For the input files, it says "We store the datasets in a single object in Amazon S3. The file is in sequence file format with block level LZO compression. The sequence file key is the row number of the dataset stored as a LongWritable and the value is the raw data stored as TextWritable."</p>
  
  <p>What do I need to do in order to process these input files with Hadoop Streaming?</p>
  
  <p>I tried adding an extra "-inputformat SequenceFileAsTextInputFormat" to my arguments, but this doesn't seem to work -- my jobs keep failing for some unspecified reason. Are there other arguments I'm missing?</p>
  
  <p>I've tried using a very simple identity as both my mapper and reducer</p>
  
  <pre><code>#!/usr/bin/env ruby
  
  STDIN.each do |line|
    puts line
  end
  </code></pre>
  
  <p>but this doesn't work.</p>