Create PySpark Profile for IPython
Question: I follow this link <http://ramhiser.com/2015/02/01/configuring-ipython-
notebook-support-for-pyspark/> in order to create PySpark Profile for IPython.

    
    
    00-pyspark-setup.py
    # Configure the necessary Spark environment
    import os
    import sys
    
    spark_home = os.environ.get('SPARK_HOME', None)
    sys.path.insert(0, spark_home + "\python")
    
    # Add the py4j to the path.
    # You may need to change the version number to match your install
    sys.path.insert(0, os.path.join(spark_home, '\python\lib\py4j-0.8.2.1-src.zip'))
    
    # Initialize PySpark to predefine the SparkContext variable 'sc'
    execfile(os.path.join(spark_home, '\python\pyspark\shell.py'))
    

My problem when I type `sc` in ipython-notebook, I got `''` I should see
output similar to `<pyspark.context.SparkContext at 0x1097e8e90>.`

Any idea about how to resolve it ?


Answer: I was trying to do the same, but had problems. Now, I use `findspark`
(<https://github.com/minrk/findspark>) instead. You can install it with pip
(see <https://pypi.python.org/pypi/findspark/>):

    
    
    $ pip install findspark
    

And then, inside a notebook:

    
    
    import findspark
    findspark.init()
    
    import pyspark
    sc = pyspark.SparkContext(appName="myAppName")
    

If you want to avoid this boilerplate, you can put the above 4 lines in
`00-pyspark-setup.py`.

(Right now I have Spark 1.4.1. and findspark 0.0.5.)

