title: How to integrate Jupyter notebook scala kernel with apache spark?
body: <p>I have installed Scala kernel based on this doc: <a href="https://github.com/jupyter-scala/jupyter-scala" rel="nofollow noreferrer">https://github.com/jupyter-scala/jupyter-scala</a>
  Kernel is there:</p>
  
  <pre><code>$ jupyter kernelspec list
  Available kernels:
    python3     /usr/local/homebrew/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ipykernel/resources
    scala       /Users/bobyfarell/Library/Jupyter/kernels/scala
  </code></pre>
  
  <p>When I try to use Spark in the notebook I get this:</p>
  
  <pre><code>val sparkHome = "/opt/spark-2.3.0-bin-hadoop2.7"
  val scalaVersion = scala.util.Properties.versionNumberString
  import org.apache.spark.ml.Pipeline
  
  Compilation Failed
  Main.scala:57: object apache is not a member of package org
   ; import org.apache.spark.ml.Pipeline
                ^
  </code></pre>
  
  <p>I tried:</p>
  
  <ul>
  <li>Setting SPARK_HOME and CLASSPATH to the location of $SPARK_HOME/jars</li>
  <li>Setting -cp option pointing to $SPARK_HOME/jars in kernel.json</li>
  <li>Setting classpath.add call before imports</li>
  </ul>
  
  <p>None of these helped. Please note I don't want to use Toree, I want to use standalone spark and Scala kernel with Jupyter. A similar issue is reported here too: <a href="https://github.com/jupyter-scala/jupyter-scala/issues/63" rel="nofollow noreferrer">https://github.com/jupyter-scala/jupyter-scala/issues/63</a></p>