How do I set/get heap size for Spark (via Python notebook)
You can manage Spark memory limits programmatically (by the API). 

As SparkContext is already available in your Notebook:

sc._conf.get('spark.driver.memory')


You can set as well, but you have to shutdown the existing SparkContext first:

conf = SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
        .set('spark.executor.memory', '4G')
        .set('spark.driver.memory', '45G')
        .set('spark.driver.maxResultSize', '10G'))
sc = SparkContext(conf=conf)


If your workload is the same for all analysis, then editing spark-defaults.conf as cited above is the way to go.