title: How to write Spark logs to S3 when using Kubernetes (EKS)? Error: Unrecognized option: --spark.kubernetes.driver.secretKeyRef.AWS_ACCESS_KEY_ID
body: <p>I'm trying to submit a Spark job on Kubernetes and write logs to S3. I'm using EKS and Spark client mode</p>
  
  <p>I can write my Spark logs to a local directory, e.g., the below works:</p>
  
  <pre><code>bin/spark-submit \
  --master k8s://https://&lt;k8s-master&gt;:443 \
  --deploy-mode client \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=5 \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=file:/tmp/spark_data \
  --conf spark.kubernetes.container.image=gcav66/spark:spark-gus \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  file:///home/gus/spark-2.4.4-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.4.jar 
  </code></pre>
  
  <p>I'd like to store the logs in an S3 bucket. Based on the Spark docs (<a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#secret-management" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html#secret-management</a>), I'm trying to pass my AWS Access and Secret keys via <code>spark.kubernetes.driver.secretKeyRef.[EnvName]</code></p>
  
  <pre><code>bin/spark-submit \
  --master k8s://https://&lt;k8s-master&gt;:443 \
  --deploy-mode client \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=3 \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=s3a://&lt;my-bucket&gt; \
  --conf spark.history.fs.logDirectory=s3a://&lt;my-bucket&gt; \
  --conf spark.kubernetes.container.image=gcav66/spark:spark-gus \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --spark.kubernetes.driver.secretKeyRef.AWS_ACCESS_KEY_ID=aws-secrets:key \
  --spark.kubernetes.driver.secretKeyRef.AWS_SECRET_ACCESS_KEY=aws-secrets:secret \
  --spark.kubernetes.executor.secretKeyRef.AWS_ACCESS_KEY_ID=aws-secrets:key \
  --spark.kubernetes.executor.secretKeyRef.AWS_SECRET_ACCESS_KEY=aws-secrets:secret \
  file:///home/gus/spark-2.4.4-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.4.jar 
  </code></pre>
  
  <p>The error I get is:
  <code>Error: Unrecognized option: --spark.kubernetes.driver.secretKeyRef.AWS_ACCESS_KEY_ID</code></p>
  
  <p>I have both an environment variable <code>AWS_ACCESS_KEY_ID</code> and I created a Kubernetes secret via <code>kubectl create secret generic aws-secrets --from-file=aws-access-key --from-file=aws-secret-key</code></p>
  
  <pre><code>$ kubectl get secret
  NAME                                                    TYPE                                  DATA   AGE
  aws-secrets                                             Opaque                                2      8h
  </code></pre>
  
  <p>I don't think I have a typo - any ideas?</p>