title: Running a pyspark program on python3 kernel in jupyter notebook
body: <p>I used <code>pip install pyspark</code> to install PySpark. I didn't set any path etc.; however, I found that everything was downloaded and copied into <code>C:/Users/Admin/anaconda3/scripts</code>. I opened jupyter notebook in a Python3 kernel and I tried to run a SystemML script but it was giving me an error. I realized that I needed to place winutils.exe in <code>C:/Users/Admin/anaconda3/scripts</code> as well, so I did that and the script ran as expected.</p>
  <p>Now, my program includes GridSearch and when I run it on my personal laptop, it is markedly slower than how it is on a Cloud data platform where I can initiate a kernel with Spark (such as IBM Watson Studio).</p>
  <p>So my questions are:</p>
  <p>(i) How do I add PySpark to the Python3 kernel? Or is it already working in the background when I <code>import pyspark</code>?</p>
  <p>(ii) When I run the same code on the same dataset using pandas and scikit-learn, there is not much difference in performance. When is PySpark preferred/beneficial over pandas and scikit-learn?</p>
  <p>Another thing is, even though PySpark seems to be working fine and I'm able to import its libraries, when I try to run</p>
  <pre><code>import findspark
  findspark.init()
  </code></pre>
  <p>it throws up and error (on line 2), saying the <code>list is out of range</code>. I googled a bit and found an advice that said that I had to explicitly set <code>SPARK_HOME='C:/Users/Admin/anaconda3/Scripts'</code>; but when I do that, pyspark stops working (findspark.init() still not working).</p>
  <p>If anyone can explain what is going on, I'd be very grateful. Thank you.</p>