title: Zeppelin Notebook %pyspark interpreter vs %python interpreter
body: <p>What's the difference between <code>%python</code> and <code>%pyspark</code> in a Zeppelin notebook (screenshot below)?</p>
  
  <ul>
  <li>I can run the same python commands with both cases (like <code>print('hello')</code>)
  
  <ul>
  <li><strike>I can use the the same PySpark API in both cases</strike></li>
  <li><strike>i.e. <code>from pyspark.sql import SparkSession</code>, and <code>spark.read.csv</code></strike></li>
  <li><strong>EDIT 10/31/2019</strong> This is no longer true; in a <code>%python</code> interpreter I get the message <code>No module named pyspark</code>. </li>
  <li>I guess I can install the missing module using <code>pip install pyspark</code>, but I <strong>don't know how to install onto a Zeppelin resource.</strong> </li>
  </ul></li>
  <li><p><strong>EDIT 10/31/2019</strong> I must use a <code>python</code> interpreter, vs a <code>python3</code> interpreter, else I get an error like:
  <code>Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.</code></p>
  
  <ul>
  <li>Also, I guess this module was installed on Zeppelin when I used it earlier this year.</li>
  </ul></li>
  <li><strike>I can even toggle back and forth; using them both <em>simultaneously</em>?</strike>
  
  <ul>
  <li><strike>i.e. first paragraph uses <code>%python</code>, next paragraph uses <code>%pyspark</code></strike></li>
  <li>Nevermind; each language cannot see the variables defined by the other language... </li>
  <li>They just have the same (Python) API, i.e. each can create their own dataframe <code>spark.createDataFrame([...])</code></li>
  </ul></li>
  <li>I see from the screenshot below that those languages use different
  <em>interpeters</em>: 
  
  <ul>
  <li><code>%python</code> language  -> <code>python</code> interpreter </li>
  <li><code>%pyspark</code> language -> <code>spark</code> interpreter </li>
  </ul></li>
  </ul>
  
  <p>...But what's the difference between using those interpreters, if my API / code is all the same? Is either of them faster/newer/better? Why use one over the other?</p>
  
  <p><a href="https://i.stack.imgur.com/Qx3Iw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Qx3Iw.png" alt="a zeppelin notebook settings dialog, showing 3 interpreters (spark, python, md) and in the spark interpreter a list of languages including %pyspark, and in python interpreter a list of languages including %python"></a></p>