Anaconda + Spark - changing python version for ipython notebooks
Question: I installed Anaconda, and try to work with spark on top. When I launch spark
with Ipython_OPTS="notebook", the python version is the one associated with
anaconda's version of python for the notebook.

    
    
    $ conda search python
    Using Anaconda Cloud api site https://api.anaconda.org
    Fetching package metadata: ....
    ipython                      0.13                     py26_0  defaults
                              *  4.1.2                    py35_1  defaults
    ipython-notebook             0.13.2                   py27_0  defaults
                                 4.0.4                    py27_0  defaults
                                 4.0.4                    py34_0  defaults
                                 4.0.4                    py35_0  defaults
    python                       1.0.1                         0  defaults
                              .  2.7.11                        0  defaults
                              *  3.5.1                         0  defaults
    

And if start spark-shell I can precise the environment version on which I am
interested (I want 2.7.11) :

    
    
    $ PYSPARK_PYTHON=/Applications/anaconda/anaconda/envs/vingt-sept/bin/python  pyspark
    Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec  6 2015, 18:57:58)
    

but if I start spark with the ipython notebook then it defaults back to the
python 3.5 version :-(

How can I link the default ipython version to the same version as the one of
my env "vingt-sept" ?


Answer: Similar to how you are setting the `PYSPARK_PYTHON` environment variable for
the `pyspark` shell, you can also set this environment variable in your
IPython/Jupyter notebook using:

    
    
    import os
    os.environ["PYSPARK_PYTHON"] = "/Applications/anaconda/anaconda/envs/vingt-sept/bin/python"
    

Refer to [this blog post](https://www.continuum.io/blog/developer-blog/using-
anaconda-pyspark-distributed-language-processing-hadoop-cluster) for more
information about setting `PYSPARK_PYTHON` and other Spark-related environment
variables from your notebook.

