title: spark on yarn and --archives option
body: <p>I am trying to utilize the --archives option available in spark-on-yarn in order to upload an archive file. Based on the documentation &amp; as mentioned in this <a href="https://stackoverflow.com/questions/41498365/upload-zip-file-using-archives-option-of-spark-submit-on-yarn">question</a>, yarn will not only upload the zip file but will also automatically unarchive the zip file on the worker nodes.</p>
  
  <p>From the logs, I can see that yarn is uploading the jar in spark's staging directory e.g.</p>
  
  <pre><code>17/09/19 01:28:57 INFO Client: Uploading resource file:/home/foo/bar/zoo.zip -&gt; hdfs://abc.foo.bar:8020/user/xyz/.sparkStaging/application_1503584958553_4501/zoo.zip
  </code></pre>
  
  <p>The issue I am facing is that, although the zip file is getting copied into spark staging directory, it's not getting automatically unarchived &amp; I am guessing it's also not getting copied in the worker nodes.</p>
  
  <p>Assuming yarn does unarchive the zip files, is there a way to access the location of worker nodes programmatically?</p>
  
  <p>I am running spark 2.2 against emr 5.8 which is having yarn 2.7.</p>