title: spark-submit failing to connect to metastore due to Kerberos : Caused by GSSException: No valid credentials provided . but works in local-client mode
body: <p>it seems, in docker pyspark shell in local-client mode is working and able to connect to hive. However, issuing spark-submit with all dependencies it fails with below error.</p>
  <pre><code>20/08/24 14:03:01 INFO storage.BlockManagerMasterEndpoint: Registering block manager test.server.com:41697 with 6.2 GB RAM, BlockManagerId(3, test.server.com, 41697, None)
  20/08/24 14:03:02 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
  20/08/24 14:03:02 INFO hive.metastore: Trying to connect to metastore with URI thrift://metastore.server.com:9083
  20/08/24 14:03:02 ERROR transport.TSaslTransport: SASL negotiation failure
  javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
          at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
          at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
          at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
  </code></pre>
  <p>Running a simple pi example on pyspark works fine with no kerberos issues, but when trying to access hive getting kerberos error.</p>
  <p>Spark-submit command:</p>
  <pre><code>spark-submit --master yarn --deploy-mode cluster --files=/etc/hive/conf/hive-site.xml,/etc/hive/conf/yarn-site.xml,/etc/hive/conf/hdfs-site.xml,/etc/hive/conf/core-site.xml,/etc/hive/conf/mapred-site.xml,/etc/hive/conf/ssl-client.xml  --name fetch_hive_test --executor-memory 12g --num-executors 20 test_hive_minimal.py
  </code></pre>
  <p>test_hive_minimal.py is a simple pyspark script to show tables in test db:</p>
  <pre><code>from pyspark.sql import SparkSession
  #declaration
  appName = &quot;test_hive_minimal&quot;
  master = &quot;yarn&quot;
  # Create the Spark session
  sc = SparkSession.builder \
      .appName(appName) \
      .master(master) \
      .enableHiveSupport() \
      .config(&quot;spark.hadoop.hive.enforce.bucketing&quot;, &quot;True&quot;) \
      .config(&quot;spark.hadoop.hive.support.quoted.identifiers&quot;, &quot;none&quot;) \
      .config(&quot;hive.exec.dynamic.partition&quot;, &quot;True&quot;) \
      .config(&quot;hive.exec.dynamic.partition.mode&quot;, &quot;nonstrict&quot;) \
      .getOrCreate()
  # Define the function to load data from Teradata
  #custom freeform query
  sql = &quot;show tables in user_tables&quot;
  df_new = sc.sql(sql)
  df_new.show()
  sc.stop()
  </code></pre>
  <p>Can anyone throw some light how to fix this? Isnt kerberos tickets managed automatically by yarn? all other hadoop resources are accessible.</p>
  <p><strong>UPDATE:</strong>
  <strong>Issue was fixed after sharing vol mount on the docker container and passing keytab/principal along with hive-site.xml for accessing metastore.</strong></p>
  <pre><code>spark-submit --master yarn \
  --deploy-mode cluster \
  --jars /srv/python/ext_jars/terajdbc4.jar \
  --files=/etc/hive/conf/hive-site.xml \
  --keytab /home/alias/.kt/alias.keytab \ #this is mounted and kept in docker local path 
  --principal alias@realm.com.org \
  --name td_to_hive_test \
  --driver-cores 2 \
  --driver-memory 2G \
  --num-executors 44 \
  --executor-cores 5 \
  --executor-memory 12g \
  td_to_hive_test.py
  </code></pre>