Use spark_csv inside Jupyter and using Python
Assuming the rest of your configuration is correct all you have to do is to make spark-csv jar available to your program. There are a few ways you can achieve this:


manually download required jars including  spark-csv and csv parser (for example org.apache.commons.commons-csv) and put them somewhere on the CLASSPATH.
using --packages option (use Scala version which has been used to build Spark. Pre-built versions use 2.10):


using PYSPARK_SUBMIT_ARGS environmental variable:

export PACKAGES="com.databricks:spark-csv_2.11:1.3.0"
export PYSPARK_SUBMIT_ARGS="--packages ${PACKAGES} pyspark-shell"

adding Gradle string to spark.jars.packages in conf/spark-defaults.conf:

spark.jars.packages    com.databricks:spark-csv_2.11:1.3.0