title: Initialize pyspark in jupyter notebook using the spark-defaults.conf file
body: <p>There are few different answers for instantiating pyspark on the web and SO. however some are outdated and some do not cover all the "desired and optimum"* characters of starting spark. Which is squeezing and making available every bit of RAM and CPU you have for the spark session. Thus, I am trying to figure out a way to do the following.</p>
  
  <p>1) Utilize the maximum number of cores
  <br>
  2) Utilize the maximum number of driver memory
  <br>
  3) Utilize the maximum number of executor memory
  <br>
  4) Utilize the maximum stack size and heap size.
  <br>
  5) Passing the maximum size to spark.driver.extraJavaOptions and spark.executor.extraJavaOption
  <br>
  6) Use the Kryo.serializer recommended in highly reviewed books on spark in 2017 - 2018.</p>
  
  <p>And any other properties seen appropriate.</p>
  
  <p>A generalized approach will instantiate a spark session. The purpose of this spark session is to create a DataFrame from a DataBase later.</p>
  
  <p>However, I could not find a way that combines all of these steps into a smart and working way. Some suggest a <code>SparkSession</code>, Some suggest passing conf in <code>SparkConf</code>, some suggest passing it in the shell command.</p>
  
  <p>Here is what I tried so far:</p>
  
  <pre><code>spark = SparkSession.builder.appName('ose')\
      .master('local[*]')\
      .config('spark.driver.memory', '32g')\
      .config('spark.executor.memory', '32g')\
      .config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer')\
      .getOrCreate()
  
  df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri",
  "mongodb://localhost/demo.demo_lab").load()
  </code></pre>
  
  <p>The virtual system properties is</p>
  
  <pre><code>Spark Version: '2.3.1'
  Scala Version: version 2.11.8
  Python Version: 3.6.6 | Anaconda custom (64-bit)
  Java Version: 1.8.0_181 (Oracle Corporation)
  
  
  Linux: Ubuntu 16.04.5 LTS
  CPU(s):                8
  On-line CPU(s) list:   0-7
  Thread(s) per core:    1
  Core(s) per socket:    8
  Socket(s):             1
  NUMA node(s):          1
  Model name:            Intel(R) Core(TM) i5-6700 CPU @ 2.00GHz
  </code></pre>
  
  <p>Furthermore, I am working on a jupyter notebook in my local computer. The <a href="https://spark.apache.org/docs/latest/configuration.html" rel="nofollow noreferrer">documentation</a> mentions for some properties that </p>
  
  <blockquote>
    <p>Note: In client mode, this config must not be set through the
    SparkConf directly in your application, because the driver JVM has
    already started at that point. Instead, please set this through the
    --driver-memory command line option or in your default properties file.</p>
  </blockquote>
  
  <p>It is not possible to use command line, as I am submitting my jobs through the notebook. Thus, I will stick to configuring the default properties file for all the properties just to create one entry point to maximize the properties for spark. Then instantiate spark from the Jupiter notebook.</p>
  
  <p>My question is, how to change the <code>spark-defaults.conf.template</code> file to squeeze all the juice out of the virtual computer just mentioned above for the spark session then correctly instantiate the spark session in the jupyter notebook and check that the properties have been passed in the notebook ?</p>
  
  <p>The spark-defaults file is located in.</p>
  
  <pre><code>/opt/apache-spark/spark-2.3.1-bin-hadoop2.7/conf/spark-defaults.conf.template
  </code></pre>
  
  <p>*highly opinionated point of view.</p>