title: Link Spark with iPython Notebook
body: <p>I have followed some tutorial online but they do not work with <code>Spark 1.5.1</code> on OS X El Capitan (10.11)</p>
  
  <p>Basically I have run this commands download <code>apache-spark</code></p>
  
  <pre><code>brew update
  brew install scala
  brew install apache-spark
  </code></pre>
  
  <p>updated the .bash_profile</p>
  
  <pre><code># For a ipython notebook and pyspark integration
  if which pyspark &gt; /dev/null; then
    export SPARK_HOME="/usr/local/Cellar/apache-spark/1.5.1/libexec/"
    export PYSPARK_SUBMIT_ARGS="--master local[2]"
  fi
  </code></pre>
  
  <p>run </p>
  
  <pre><code>ipython profile create pyspark
  </code></pre>
  
  <p>created a startup file <code>~/.ipython/profile_pyspark/startup/00-pyspark-setup.py</code> configured in this way </p>
  
  <pre><code># Configure the necessary Spark environment
  import os
  import sys
  
  # Spark home
  spark_home = os.environ.get("SPARK_HOME")
  
  # If Spark V1.4.x is detected, then add ' pyspark-shell' to
  # the end of the 'PYSPARK_SUBMIT_ARGS' environment variable
  spark_release_file = spark_home + "/RELEASE"
  if os.path.exists(spark_release_file) and "Spark 1.4" in open(spark_release_file).read():
      pyspark_submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "")
      if not "pyspark-shell" in pyspark_submit_args: pyspark_submit_args += " pyspark-shell"
      os.environ["PYSPARK_SUBMIT_ARGS"] = pyspark_submit_args
  
  # Add the spark python sub-directory to the path
  sys.path.insert(0, spark_home + "/python")
  
  # Add the py4j to the path.
  # You may need to change the version number to match your install
  sys.path.insert(0, os.path.join(spark_home, "python/lib/py4j-0.8.2.1-src.zip"))
  
  # Initialize PySpark to predefine the SparkContext variable 'sc'
  execfile(os.path.join(spark_home, "python/pyspark/shell.py"))
  </code></pre>
  
  <p>I then run <code>ipython notebook --profile=pyspark</code> and the notebook works fine, but the <code>sc</code> (spark context) is not recognised.</p>
  
  <p>Anyone managed to do this with <code>Spark 1.5.1</code>?</p>
  
  <p><strong>EDIT:</strong> you can follow this guide to have it working </p>
  
  <p><a href="https://gist.github.com/tommycarpi/f5a67c66a8f2170e263c">https://gist.github.com/tommycarpi/f5a67c66a8f2170e263c</a></p>