title: Fail to enable hive support in Spark submit (spark 3)
body: <p>I am using spark in an integration test suite. It has to run locally and read/write files to local file-system. I also want to read/write these data as tables.</p>
  <p>In the first step of the suite I write some hive tables in the db <code>feature_store</code> specifying<br />
  <code>spark.sql.warehouse.dir=/opt/spark/work-dir/warehouse</code>. The step completes correctly and I see the files in the folder I expect.</p>
  <p>Afterwards I run a spark-submit step with (among others) these confs</p>
  <pre><code>--conf spark.sql.warehouse.dir=/opt/spark/work-dir/warehouse --conf spark.sql.catalogImplementation=hive
  </code></pre>
  <p>and when trying to read a table previously written I get</p>
  <pre><code> Exception in thread &quot;main&quot; org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'feature_store' not found
  </code></pre>
  <p>However if I try to do exactly the same thing with exactly the same configs in a spark-shell I am able to read the data.</p>
  <p>In the spark-submit I use the following code to get the spark-session</p>
  <pre><code>SparkSession spark = SparkSession.active();
  </code></pre>
  <p>I have also tried to use instead</p>
  <pre><code>SparkSession spark = SparkSession.builder().enableHiveSupport().getOrCreate();
  </code></pre>
  <p>but I keep getting the same problem as above.</p>
  <p>I have understood that the problem is related to the spark-submit not picking up hive as
  catalog implementation. In fact I see that the class <code>spark.catalog</code> is not an instance of <code>HiveCatalogImpl</code> during the spark-submit (while it is when using spark-shell).</p>