title: Hadoop Streaming: Mapper 'wrapping' a binary executable
body: <p>I have a pipeline that I currently run on a large university computer cluster.  For publication purposes I'd like to convert it into mapreduce format such that it could be run by anyone on using a hadoop cluster such as amazon webservices (AWS).  The pipeline currently consists of as series of python scripts that wrap different binary executables and manage the input and output using the python subprocess and tempfile modules.  Unfortunately I didn’t write the binary executables and many of them either don’t take STDIN or don't emit STDOUT in a ‘useable’ fashion (e.g., only sent it to files).  These problems are why I’ve wrapped most of them in python.</p>
  
  <p>So far I’ve been able to modify my Python code such that I have a mapper and a reducer that I can run on my local machine in the standard ‘test format.’</p>
  
  <pre><code>$ cat data.txt | mapper.py | reducer.py
  </code></pre>
  
  <p>The mapper formats each line of data the way the binary it wraps wants it, sends the text to the binary using subprocess.popen (this also allows me to mask a lot of spurious STDOUT), then collects the STOUT I want, and formats it into lines of text appropriate for the reducer.
  The problems arise when I try to replicate the command on a local hadoop install.  I can get the mapper to execute, but it give an error that suggests that it can’t find the binary executable.  </p>
  
  <blockquote>
    <p>File
    "/Users/me/Desktop/hadoop-0.21.0/./phyml.py",
    line 69, in 
        main()   File "/Users/me/Desktop/hadoop-0.21.0/./mapper.py",
    line 66, in main
        phyml(None)   File "/Users/me/Desktop/hadoop-0.21.0/./mapper.py",
    line 46, in phyml
        ft = Popen(cli_parts, stdin=PIPE, stderr=PIPE, stdout=PIPE)   File
    "/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/subprocess.py",
    line 621, in <strong>init</strong>
        errread, errwrite)   File "/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/subprocess.py",
    line 1126, in _execute_child
        raise child_exception
    OSError: [Errno 13] Permission denied</p>
  </blockquote>
  
  <p>My hadoop command looks like the following:</p>
  
  <pre><code>./bin/hadoop jar /Users/me/Desktop/hadoop-0.21.0/mapred/contrib/streaming/hadoop-0.21.0-streaming.jar \
  -input /Users/me/Desktop/Code/AWS/temp/data.txt \
  -output /Users/me/Desktop/aws_test \
  -mapper  mapper.py \
  -reducer  reducer.py \
  -file /Users/me/Desktop/Code/AWS/temp/mapper.py \
  -file /Users/me/Desktop/Code/AWS/temp/reducer.py \
  -file /Users/me/Desktop/Code/AWS/temp/binary
  </code></pre>
  
  <p>As I noted above it looks to me like the mapper isn't aware of the binary - perhaps it's not being sent to the compute node?  Unfortunately I can't really tell what the problem is. Any help would be greatly appreciated. It would be particulary nice to see some hadoop streaming  mappers/reducers written in python that wrap binary executables. I can’t imagine I’m the first one to try to do this!  In fact, here is another post asking essentially the same question, but it hasn't been answered yet...</p>
  
  <p><a href="https://stackoverflow.com/questions/4101815/hadoop-elastic-map-reduce-with-binary-executable">Hadoop/Elastic Map Reduce with binary executable?</a></p>