title: Managing python dependencies with spark on google cloud dataproc
body: <p>I have written a Spark script that depends on <code>six</code> and various other python packages.</p>
  
  <pre><code>$ cat ./test_package/__init__.py
  from six.moves.urllib.request import urlopen
  
  def download_size(url):
      return len(urlopen(url).read())
  </code></pre>
  
  <p>As such, I have written a <code>setup.py</code> which states such dependencies.</p>
  
  <pre><code>$ cat ./setup.py
  from setuptools import setup
  setup(
      name="Test App",
      packages=['test_package'],
      version="0.1",
      install_requires=['six&gt;=1.0'],
  </code></pre>
  
  <p>Then in my spark script, I have code which requires the package</p>
  
  <pre><code>$ cat spark_script.py
  #!/usr/lib/spark/bin/spark-submit
  from pyspark import SparkContext
  from glob import glob
  from test_package import download_size
  
  sc = SparkContext()
  sc.addPyFile(glob('dist/Test_App-*.egg')[0])
  
  ...
  
  sc.parallelize(urls).map(download_size).collect()
  </code></pre>
  
  <p>If I run</p>
  
  <pre><code>$ ./test.py
  </code></pre>
  
  <p>It works fine. However, if I try to use python3,</p>
  
  <pre><code>$ PYSPARK_PYTHON=python3 ./test.py
  </code></pre>
  
  <p>The master node is able to import <code>test_package</code>, but in the middle of the mapreduce I get this on each worker node:</p>
  
  <pre><code>  File "/hadoop/yarn/nm-local-dir/usercache/sam/appcache/application_1487279780844_0041/container_1487279780844_0041_01_000003/pyspark.zip/pyspark/serializers.py", line 419, in loads
      return pickle.loads(obj, encoding=encoding)
    File "./Test_App-0.1-py2.7.egg/test_package/__init__.py", line 2, in &lt;module&gt;
      from six.moves.urllib.request import urlopen
  ImportError: No module named 'six'
  </code></pre>
  
  <p>How do I manage python dependencies on an Google cloud dataproc provisioned Apache spark cluster?</p>