title: how to set spark.sql.shuffle.partitions when using the lastest spark version
body: <p>I want to reset the spark.sql.shuffle.partitions configure in the pyspark code, since I need to join two big tables. But the following code doesn't not work in the latest spark version, the error says that "no method "setConf" in xxx"</p>
  
  <pre><code>#!/usr/bin/python
  # -*- coding: utf-8 -*-
  import sys
  import pyspark
  from pyspark.context import SparkContext
  from pyspark.sql.session import SparkSession
  sc = SparkContext('local')
  spark = SparkSession(sc)
  
  spark.sparkContext.setConf("spark.sql.shuffle.partitions", "1000")
  spark.sparkContext.setConf("spark.default.parallelism", "1000")
  
  # or using the follow, neither is working 
  spark.setConf("spark.sql.shuffle.partitions", "1000")
  spark.setConf("spark.default.parallelism", "1000")
  </code></pre>
  
  <p>I would like to know how to reset the "spark.sql.shuffle.partitions" now.</p>