title: Unable to use an existing Hive permanent UDF from Spark SQL
body: <p>I have previously registered a UDF with hive. It is permanent not <code>TEMPORARY</code>. It works in beeline.</p>
  
  <pre><code>CREATE FUNCTION normaliseURL AS 'com.example.hive.udfs.NormaliseURL' USING JAR 'hdfs://udfs/hive-udfs.jar';
  </code></pre>
  
  <p>I have spark configured to use the hive metastore. The config is working as I can query hive tables. I can see the UDF;</p>
  
  <pre><code>In [9]: spark.sql('describe function normaliseURL').show(truncate=False)
  +-------------------------------------------+
  |function_desc                              |
  +-------------------------------------------+
  |Function: default.normaliseURL             |
  |Class: com.example.hive.udfs.NormaliseURL  |
  |Usage: N/A.                                |
  +-------------------------------------------+
  </code></pre>
  
  <p>However I cannot use the UDF in a sql statement;</p>
  
  <pre><code>spark.sql('SELECT normaliseURL("value")')
  AnalysisException: "Undefined function: 'default.normaliseURL'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7"
  </code></pre>
  
  <p>If I attempt to register the UDF with spark (bypassing the metastore) it fails to register it, suggesting that it does already exist.</p>
  
  <pre><code>In [12]: spark.sql("create function normaliseURL as 'com.example.hive.udfs.NormaliseURL'")
  AnalysisException: "Function 'default.normaliseURL' already exists in database 'default';"
  </code></pre>
  
  <p>I'm using Spark 2.0, hive metastore 1.1.0. The UDF is scala, my spark driver code is python.</p>
  
  <p>I'm stumped.</p>
  
  <ul>
  <li>Am I correct in my assumption that Spark can utilise metastore-defined permanent UDFs?</li>
  <li>Am I creating the function correctly in hive?</li>
  </ul>