title: PySpark (Step/Job) on EMR cannot connect to AWS Glue Data Catalog but Zeppelin can
body: <p>I have setup an EMR cluster with Data Catalog enabled</p>
  
  <p><a href="https://i.stack.imgur.com/0blh4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0blh4.png" alt="enter image description here"></a></p>
  
  <p>I can access the data catalog when I use Zeppelin, but with jobs/steps I submit like: </p>
  
  <pre><code>aws emr add-steps --cluster-id j-XXXXXX --steps "Type=spark,Name=Test,Args=[--deploy-mode,cluster,--master,yarn,--conf,spark.yarn.submit.waitAppCompletion=false,--num-executors,2,--executor-cores,2,--executor-memory,8g,s3://XXXXXX/emr-test.py],ActionOnFailure=CONTINUE"
  </code></pre>
  
  <p>I cannot see my data catalog when I use <code>spark.sql("USE xxx")</code> OR <code>spark.sql("SHOW DATABASES")</code> why is that. </p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark.sql import SparkSession
  
  sc = SparkContext()
  spark = SparkSession \
      .builder \
      .appName("Test") \
      .config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory") \
      .getOrCreate()
  
  spark.sql("USE ...")
  spark.sql("SHOW TABLES").show()
  spark.sql("SELECT querydatetime FROM flights LIMIT 10").show(10)
  
  sc.stop()
  </code></pre>
  
  <p>I get something like: </p>
  
  <pre><code>pyspark.sql.utils.AnalysisException: u"Database 'xxxxxx' not found;"
  </code></pre>