title: custom spark does not find hive databases when running on yarn
body: <p>Starting a custom version of spark on yarn in HDP works fine following the tutorial from <a href="https://georgheiler.com/2019/05/01/headless-spark-on-yarn/" rel="nofollow noreferrer">https://georgheiler.com/2019/05/01/headless-spark-on-yarn/</a> i.e. following:</p>
  
  <pre><code># download a current headless version of spark
  export SPARK_DIST_CLASSPATH=$(hadoop classpath)
  export HADOOP_CONF_DIR=/usr/hdp/current/spark2-client/conf
  export SPARK_HOME=&lt;&lt;path/to&gt;&gt;/spark-2.4.3-bin-without-hadoop/
  &lt;&lt;path/to&gt;&gt;/spark-2.4.3-bin-without-hadoop/bin/spark-shell --master yarn --deploy-mode client --queue &lt;&lt;my_queue&gt;&gt; --conf spark.driver.extraJavaOptions='-Dhdp.version=2.6.&lt;&lt;version&gt;&gt;' --conf spark.yarn.am.extraJavaOptions='-Dhdp.version=2.6.&lt;&lt;version&gt;&gt;'
  
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 2.4.3
        /_/
  </code></pre>
  
  <p>However, a:</p>
  
  <pre><code>spark.sql("show databases").show
  </code></pre>
  
  <p>only returns:</p>
  
  <pre><code>+------------+
  |databaseName|
  +------------+
  |     default|
  +------------+
  </code></pre>
  
  <p>Now trying to pass the original HDP configuration (which is apparently not read in by my custom version of spark) like:</p>
  
  <p>one:</p>
  
  <pre><code>--files /usr/hdp/current/spark2-client/conf/hive-site.xml
  </code></pre>
  
  <p>two:</p>
  
  <pre><code>--conf spark.hive.metastore.uris='thrift://master001.my.corp.com:9083,thrift://master002.my.corp.com:9083,thrift://master003.my.corp.com:9083' --conf spark.hive.metastore.sasl.enabled='true' --conf hive.metastore.uris='thrift://master001.my.corp.com:9083,thrift://master002.my.corp.com:9083,thrift://master003.my.corp.com:9083' --conf hive.metastore.sasl.enabled='true'
  </code></pre>
  
  <p>three:</p>
  
  <pre><code>--conf spark.yarn.dist.files='/usr/hdp/current/spark2-client/conf/hive-site.xml'
  </code></pre>
  
  <p>four:</p>
  
  <pre><code>--conf spark.sql.warehouse.dir='/apps/hive/warehouse'
  </code></pre>
  
  <p>all does not help to solve the issue.
  How can I get spark to recognize the hive databases?</p>