title: Hadoop and NLTK: Fails with stopwords
body: <p>I'm trying to run a Python program on Hadoop.  The program involves the NLTK library.  The program also utilizes the Hadoop Streaming API, as described <a href="http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/#run-the-mapreduce-job" rel="nofollow">here</a>.</p>
  
  <p><strong>mapper.py:</strong></p>
  
  <pre><code>#!/usr/bin/env python
  import sys
  import nltk
  from nltk.corpus import stopwords
  
  #print stopwords.words('english')
  
  for line in sys.stdin:
          print line,
  </code></pre>
  
  <p><strong>reducer.py:</strong></p>
  
  <pre><code>#!/usr/bin/env python
  
  import sys
  for line in sys.stdin:
      print line,
  </code></pre>
  
  <p><strong>Console command:</strong></p>
  
  <pre><code>bin/hadoop jar contrib/streaming/hadoop-streaming.jar \ -file /hadoop/mapper.py -mapper /hadoop/mapper.py -file /hadoop/reducer.py -reducer /hadoop/reducer.py -input /hadoop/input.txt -output /hadoop/output
  </code></pre>
  
  <p>This runs perfectly, with the output simply containing the lines of the input file.  </p>
  
  <p>However, when this line (from mapper.py): </p>
  
  <p><strong>#print stopwords.words('english')</strong></p>
  
  <p>is <em>uncommented</em>, then the program fails and says </p>
  
  <blockquote>
    <p>Job not successful. Error: # of failed Map Tasks exceeded allowed
    limit. FailedCount: 1.</p>
  </blockquote>
  
  <p>I have checked and in a standalone python program, </p>
  
  <blockquote>
    <p>print stopwords.words('english')</p>
  </blockquote>
  
  <p>works perfectly fine, and so I am absolutely stumped as to why it's causing my Hadoop program to fail.  </p>
  
  <p>I would greatly appreciate any help!  Thank you</p>