title: Spark Structure Streaming job failing in cluster mode
body: <p>I am using spark-sql-2.4.1 v in my application.</p>
  <p>While writing data on to hdfs folder I am facing this issue in spark-streaming application</p>
  <p><strong>Error:</strong></p>
  <pre><code>    yarn.Client: Deleted staging directory hdfs://dev/user/xyz/.sparkStaging/application_1575699597805_47
      20/02/24 14:02:15 ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.hadoop.security.AccessControlException: Permission denied: user= xyz, access=WRITE, inode=&quot;/tmp/hadoop-admin&quot;:admin:supergroup:drwxr-xr-x
      .
      .
      .
      Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=xyz, access=WRITE, inode=&quot;/tmp/hadoop-admin&quot;:admin:supergroup:drwxr-xr-x
              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350)
              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:251)
  </code></pre>
  <p>While writing data on to HDFS folder I am facing this issue in spark-streaming application. When I run in yarn-cluster mode I face this issue i.e.</p>
  <pre><code>    --master yarn \
      --deploy-mode cluster \
  </code></pre>
  <p>But when I run in “yarn-client” mode it runs fine i.e.</p>
  <pre><code>    --master yarn \
      --deploy-mode client \
  </code></pre>
  <p>What is the root cause of this problem?</p>
  <p>Fundamental question here, why it is trying to write in &quot;<code>/tmp/hadoop-admin/</code>&quot; instead of respective user directory i.e. <code>hdfs://qa2/user/xyz/</code>?</p>
  <p>I have come across this fix:</p>
  <p><a href="https://issues.apache.org/jira/browse/SPARK-26825" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-26825</a></p>
  <p>How can I implement it in my spark-sql application?</p>