title: Spark enable security with secure YARN Hadoop cluster
body: <p>I have an Hadoop 3.0 cluster configured with Kerberos. Everything works fine and YARN is started as well.  </p>
  
  <p>Now I wish to add Spark on top of it and make full use of Hadoop and security. To do so I use a <a href="http://spark.apache.org/downloads.html" rel="noreferrer">binary distribution of Spark 2.3</a> and modified the following.  </p>
  
  <p>In <code>spark-env.sh</code>:</p>
  
  <p><code>YARN_CONF_DIR</code>, set to the folder where my Hadoop configuration files <code>core-site.xml</code>, <code>hdfs-site.xml</code> and <code>yarn-site.xml</code> are located. </p>
  
  <p>In <code>spark-defaults.conf</code>:</p>
  
  <pre><code>spark.master                yarn
  spark.submit.deployMode     cluster
  spark.authenticate          true
  spark.yarn.principal        mysparkprincipal
  spark.yarn.keytab           mykeytabfile
  </code></pre>
  
  <p>If I understood correctly when using YARN, the secret key will be generated automatically and I don't need to manually set <a href="https://spark.apache.org/docs/latest/security.html" rel="noreferrer"><code>spark.authenticate.secret</code></a>.</p>
  
  <p>The problem I have is that the worker is complaining about the key:</p>
  
  <pre><code>java.lang.IllegalArgumentException: A secret key must be specified via the spark.authenticate.secret config
  </code></pre>
  
  <p>I also don't have any indication in the logs that Spark is using YARN or try to do anything with my hdfs volume. It's almost like Hadoop configuration files are ignored completely. I've read documentation on YARN and security for Spark but it's not very clear to me.</p>
  
  <p>My questions are:</p>
  
  <ul>
  <li>How can i be sure that Spark is using YARN</li>
  <li>Do I need to set <a href="https://spark.apache.org/docs/latest/running-on-yarn.html" rel="noreferrer"><code>spark.yarn.access.hadoopFileSystems</code></a> if I only use the server set in <code>YARN_CONF_DIR</code></li>
  <li>Is <code>LOCAL_DIRS</code> best to be set to HDFS and if yes, what is the syntax</li>
  <li>Do I need both <code>HADOOP_CONF_DIR</code> and <code>YARN_CONF_DIR</code>?</li>
  </ul>
  
  <p><strong>Edit/Add:</strong></p>
  
  <p>After looking at the source code the exception is from SASL which is not enabled for Spark so I don't understand.  </p>
  
  <p>My Hadoop has SSL enabled (<a href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html#Data_confidentiality" rel="noreferrer">Data Confidentiality</a>) and since I give Spark my server configuration maybe it requires SSL for Spark if the configuration for Hadoop has it enabled.  </p>
  
  <p>So far I am really confused about everything.</p>
  
  <ul>
  <li>It says that <a href="https://spark.apache.org/docs/latest/configuration.html#environment-variables" rel="noreferrer">environment variables need to be set</a> using the <code>spark.yarn.appMasterEnv</code>. But which one? All of them?</li>
  <li>Also it says that it is Hadoop CLIENT file that I need to have on the classpath but what properties should be present in a CLIENT file?
  I guess I can replace the XML files using <code>spark.hadoop.*</code> properties but what are the properties required for Spark to know where is my YARN cluster?</li>
  <li>Setting <code>spark.authenticate.enableSaslEncryption</code> to false seems to have no effect as the exception is still about <code>SparkSaslClient</code></li>
  </ul>
  
  <p>The exception is:</p>
  
  <pre><code>java.lang.IllegalArgumentException: A secret key must be specified via the spark.authenticate.secret config
      at org.apache.spark.SecurityManager$$anonfun$getSecretKey$4.apply(SecurityManager.scala:510)
      at org.apache.spark.SecurityManager$$anonfun$getSecretKey$4.apply(SecurityManager.scala:510)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.SecurityManager.getSecretKey(SecurityManager.scala:509)
      at org.apache.spark.SecurityManager.getSecretKey(SecurityManager.scala:551)
      at org.apache.spark.network.sasl.SparkSaslClient$ClientCallbackHandler.handle(SparkSaslClient.java:137)
      at com.sun.security.sasl.digest.DigestMD5Client.processChallenge(DigestMD5Client.java:337)
      at com.sun.security.sasl.digest.DigestMD5Client.evaluateChallenge(DigestMD5Client.java:220)
      at org.apache.spark.network.sasl.SparkSaslClient.response(SparkSaslClient.java:98)
      at org.apache.spark.network.sasl.SaslClientBootstrap.doBootstrap(SaslClientBootstrap.java:71)
      at org.apache.spark.network.crypto.AuthClientBootstrap.doSaslAuth(AuthClientBootstrap.java:115)
      at org.apache.spark.network.crypto.AuthClientBootstrap.doBootstrap(AuthClientBootstrap.java:74)
      at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:257)
      at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
      at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
      at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
      at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>