title: Using Hadoop in python to process a large csv file
body: <p>I have a huge CSV file I would like to process using Hadoop MapReduce on Amazon EMR (python). </p>
  
  <p>The file has 7 fields, however, I am only looking at the <strong>date</strong> and <strong>quantity</strong> field.</p>
  
  <pre><code> "date" "receiptId" "productId" "quantity"  "price" "posId" "cashierId"
  </code></pre>
  
  <p>Firstly, my mapper.py</p>
  
  <pre><code>import sys
  
  def main(argv):
      line = sys.stdin.readline()
      try:
          while line:
              list = line.split('\t')
  
              #If date meets criteria, add quantity to express key
                  if int(list[0][11:13])&gt;=17 and int(list[0][11:13])&lt;=19:
                      print '%s\t%s' % ("Express", int(list[3]))
              #Else, add quantity to non-express key
                  else:
                      print '%s\t%s' % ("Non-express", int(list[3]))
  
              line =  sys.stdin.readline()
  except "end of file":
          return None
  if __name__ == "__main__":
          main(sys.argv)
  </code></pre>
  
  <p>For the reducer, I will be using the streaming command: aggregate.</p>
  
  <p>Question: </p>
  
  <ol>
  <li><p>Is my code right? I ran it in Amazon EMR but i got an empty output.</p></li>
  <li><p>So my end result should be: express, XXX and non-express, YYY. Can I have it do a divide operation before returning the result? Just the result of XXX/YYY. Where should i put this code? A reducer??</p></li>
  <li><p>Also, this is a huge CSV file, so will mapping break it up into a few partitions? Or do I need to explicitly call a FileSplit? If so, how do I do that?</p></li>
  </ol>