title: Specifying python files for jupyter notebook on a Spark cluster
body: <p>I am running a jupyter-notebook on a Spark cluster (with yarn). I am using the "findspark" package to set up the notebook and it works perfectly fine (I connect to the cluster master through a SSH tunnel).
  When I write a "self-contained" notebook, it works perfectly, e.g. the following code runs with no problem:</p>
  
  <pre><code>import findspark
  findspark.init()
  
  import pyspark
  
  sc = pyspark.SparkContext(appName='myApp')
  a = sc.range(1000,numSlices=10)
  a.take(10)
  sc.stop()
  </code></pre>
  
  <p>The Spark job is perfectly distributed on the workers. However, when I want to use a python package that I wrote, <strong>the files are missing on the workers</strong>. </p>
  
  <p>When I am not using Jupyter-notebook and when I use <em>spark-submit --master yarn --py-files myPackageSrcFiles.zip</em>, my Spark job works fine, e.g. the following code runs correctly:</p>
  
  <p><strong>main.py</strong></p>
  
  <pre><code>import pyspark
  from myPackage import myFunc
  
  sc = pyspark.SparkContext(appName='myApp')
  a = sc.range(1000,numSlices=10)
  b = a.map(lambda x: myFunc(x)) 
  b.take(10)
  sc.stop()
  </code></pre>
  
  <p>Then</p>
  
  <pre><code>spark-submit --master yarn --py-files myPackageSrcFiles.zip main.py
  </code></pre>
  
  <p><strong>The question is</strong>: How to run main.py from a jupyter notebook? I tried specifying the .zip package in the SparkContext with the <em>pyfiles</em> keyword but I got an error...</p>