title: I can't seem to get --py-files on Spark to work
body: <p>I'm having a problem with using Python on Spark. My application has some dependencies, such as numpy, pandas, astropy, etc. I cannot use virtualenv to create an environment with all dependencies, since the nodes on the cluster do not have any common mountpoint or filesystem, besides HDFS. Therefore I am stuck with using <code>spark-submit --py-files</code>. I package the contents of site-packages in a ZIP file and submit the job like with <code>--py-files=dependencies.zip</code> option (as suggested in <a href="https://stackoverflow.com/questions/29495435/easiest-way-to-install-python-dependencies-on-spark-executor-nodes">Easiest way to install Python dependencies on Spark executor nodes?</a>). However, the nodes on cluster still do not seem to see the modules inside and they throw <code>ImportError</code> such as this when importing numpy.</p>
  
  <pre><code>File "/path/anonymized/module.py", line 6, in &lt;module&gt;
      import numpy
  File "/tmp/pip-build-4fjFLQ/numpy/numpy/__init__.py", line 180, in &lt;module&gt;   
  File "/tmp/pip-build-4fjFLQ/numpy/numpy/add_newdocs.py", line 13, in &lt;module&gt;
  File "/tmp/pip-build-4fjFLQ/numpy/numpy/lib/__init__.py", line 8, in &lt;module&gt;
      #
  File "/tmp/pip-build-4fjFLQ/numpy/numpy/lib/type_check.py", line 11, in &lt;module&gt;
  File "/tmp/pip-build-4fjFLQ/numpy/numpy/core/__init__.py", line 14, in &lt;module&gt;
  ImportError: cannot import name multiarray
  </code></pre>
  
  <p>When I switch to the virtualenv and use the local pyspark shell, everything works fine, so the dependencies are all there. Does anyone know, what might cause this problem and how to fix it?</p>
  
  <p>Thanks!</p>