title: PySpark 2.1: Importing module with UDF's breaks Hive connectivity
body: <p>I'm currently working with Spark 2.1 and have a main script that calls a helper module that contains all my transformation methods. In other words:</p>
  
  <pre><code>main.py
  helper.py
  </code></pre>
  
  <p>At the top of my <code>helper.py</code> file I have several custom UDFs that I have defined in the following manner:</p>
  
  <pre><code>def reformat(s):
    return reformat_logic(s)
  reformat_udf = udf(reformat, StringType())
  </code></pre>
  
  <p>Before I broke off all the UDFs into the helper file, I was able to connect to my Hive metastore through my SparkSession object using <code>spark.sql('sql statement')</code>. However, after I moved the UDFs to the helper file and imported that file at the top of my main script, the SparkSession object could no longer connect to Hive and went back to the default Derby database. I also get errors when trying to query my Hive tables such as <code>Hive support is required to insert into the following tables...</code></p>
  
  <p>I've been able to solve my issue by moving my UDFs into a completely separate file and only running the import statements for that module inside the functions that need them (not sure if this is good practice, but it works). Anyway, does anyone understand why I'm seeing such peculiar behavior when it comes to Spark and UDFs? And does anyone know a good way to share UDFs across applications?</p>