title: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
body: <p>I'm trying to connect to Kafka from spark structured streaming.</p>
  
  <p>This works:</p>
  
  <pre class="lang-sh prettyprint-override"><code>spark-shell --master local[1] \
         --files /mypath/jaas_mh.conf \
         --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 \
         --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=jaas_mh.conf" \
         --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=jaas_mh.conf" \
         --num-executors 1  --executor-cores 1 
  </code></pre>
  
  <p>However, when I try to do the same programmatically ..</p>
  
  <pre class="lang-scala prettyprint-override"><code>object SparkHelper {
    def getAndConfigureSparkSession() = {
      val conf = new SparkConf()
        .setAppName("Structured Streaming from Message Hub to Cassandra")
        .setMaster("local[1]")
        .set("spark.driver.extraJavaOptions", "-Djava.security.auth.login.config=jaas_mh.conf")
        .set("spark.executor.extraJavaOptions", "-Djava.security.auth.login.config=jaas_mh.conf")
  
      val sc = new SparkContext(conf)
      sc.setLogLevel("WARN")
  
      getSparkSession()
    }
  
    def getSparkSession() : SparkSession = {
      val spark = SparkSession
        .builder()
        .getOrCreate()
  
      spark.sparkContext.addFile("/mypath/jaas_mh.conf")
  
      return spark
    }
  }
  </code></pre>
  
  <p>I get the error:</p>
  
  <pre><code> Could not find a 'KafkaClient' entry in the JAAS configuration. 
      System property 'java.security.auth.login.config' is not set
  </code></pre>
  
  <p>Any pointers?</p>