title: Hive transform using Python: Unable to initialize custom script
body: <p>I'm trying to test Hive TRANSFORM by feeding a Python script as mapper. My hive script is:</p>
  
  <pre><code>add file  /full/path/to/mapper.py;
  
  set mapred.job.queue.name=queue_name;
  
  use my_database;
  
  select transform(s.year, s.month, s.day, s.hour) 
  using 'mapper.py' 
  from my_table s limit 10; 
  </code></pre>
  
  <p>and my Python mapper script is simply trying to echo the input:</p>
  
  <pre><code>#!/usr/local/bin/python
  import sys
  for line in sys.stdin:
      print line
  </code></pre>
  
  <p>I have tried to run this with the following combinations:</p>
  
  <ol>
  <li><p>Removing the <code>add file ...</code> in the hive script and providing full path to <code>mapper.py</code> in the <code>select ...</code> statement </p></li>
  <li><p>Keeping the <code>add file ...</code> and the full path for mapper: <code>/path/to/mapper.py</code></p></li>
  <li><p>Keeping the <code>add file ...</code> and relative path for mapper: <code>./mapper.py</code></p></li>
  <li><p>Tried selecting mapper output using <code>AS</code> clause (<code>using 'mapper.py' as line</code>)</p></li>
  </ol>
  
  <p>So far, all of the above attempts have resulted in Hive reporting that it cannot initialize my custom script:</p>
  
  <pre><code>FAILED: Execution Error, return code 20000 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Unable to initialize custom script.
  </code></pre>
  
  <p>I'm not able to understand the nature of this 'initialization.' Is Hive not able to </p>
  
  <ol>
  <li><em>find</em> my script (i.e., a path issue)? </li>
  <li>locate the python executable (i.e., the <code>#!</code> shebang) </li>
  </ol>
  
  <p>I'm following the "Custom map/reduce scripts" in the <a href="https://cwiki.apache.org/confluence/display/Hive/Tutorial#Tutorial-Custommap%2Freducescripts" rel="noreferrer">Hive tutorial</a>. </p>