title: Hive - handling NULL input values in Python UDF
body: <p>while writing <code>hive</code> <code>UDF</code>s in python, how to handle the <code>NULL</code> input values?.</p>
  
  <p>Example setup to reproduce the error:</p>
  
  <p>Hive table:</p>
  
  <pre><code>CREATE external table udfTest(
  fname STRING,
  age INT
  ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION '&lt;location&gt;';
  </code></pre>
  
  <p>input.txt:</p>
  
  <pre><code>abc    1
  efg    2
         3 //missing fname
  hij      //missing age
  </code></pre>
  
  <p>loading the data:</p>
  
  <pre><code>LOAD DATA INPATH '/tmp/input.txt' INTO TABLE udfTest;
  </code></pre>
  
  <p>Python UDF to increment the age by 1:</p>
  
  <pre><code>import sys
  import datetime
  for line in sys.stdin:
          fname, age = line.strip('\n').split('\t')
          fname = fname.upper() if (fname is not None and fname != '') else 'dummy'
          age = int(age) if (age is not None and age.strip() != 'NULL') else -1
          print '\t'.join(map(str, [fname, age]))
  </code></pre>
  
  <p>Load the UDF &amp; call it in hive:</p>
  
  <pre><code>add FILE /tmp/example.py
  SELECT TRANSFORM(fname, age) USING 'python /tmp/example.py' AS (fname, age) FROM udfTest;
  </code></pre>
  
  <p>The error, i am getting:</p>
  
  <blockquote>
    <p>Ended Job = job_1560326504017_73073 with errors Error during job,
    obtaining debugging information... Examining task ID:
    task_1560326504017_73073_m_000000 (and more) from job
    job_1560326504017_73073</p>
    
    <p>Task with the most failures(4):
    ----- Task ID:   task_1560326504017_73073_m_000000</p>
    
    <p>URL:  </p>
    
    <p>----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at
    org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:217)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)    at
    org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)   at
    org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)    at
    org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)  at
    java.security.AccessController.doPrivileged(Native Method)    at
    javax.security.auth.Subject.doAs(Subject.java:422)    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused
    by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]:
    An error occurred when trying to close the Operator running your
    custom script.    at
    org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:572)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
      at
    org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:199)
      ... 8 more</p>
    
    <p>FAILED: Execution Error, return code 20003 from
    org.apache.hadoop.hive.ql.exec.mr.MapRedTask. An error occurred when
    trying to close the Operator running your custom script.</p>
  </blockquote>
  
  <p><strong>EDITED:</strong></p>
  
  <pre><code>select * from udfTest;
  abc    1
  efg    2
         3
  hij    NULL
  </code></pre>