title: Exception: "SparkContext should only be created and accessed on the driver" while trying foreach()
body: <p>Being new to Spark, I need to read data from MySQL DB, and then update(or upsert) rows in another table based on what I've read.</p>
  <p>AFAIK, unfortunately, there's no way I can do update with DataFrameWriter, so I want to try querying directly to the DB after/while iterating over partitions.</p>
  <p>For now I'm writing a script and testing with local gluepyspark shell, Spark version 3.1.1-amzn-0.</p>
  <pre class="lang-py prettyprint-override"><code>from pyspark.context import SparkContext
  from pyspark.sql.session import SparkSession
  
  sc = SparkContext.getOrCreate()
  spark = SparkSession(sc)
  
  def f(p):
      pass
  
  sc.parallelize([1, 2, 3, 4, 5]).foreachPartition(lambda p: f(p))
  </code></pre>
  <p>When I try to import this simple code in gluepyspark shell, it raises errors saying &quot;SparkContext should only be created and accessed on the driver.&quot;</p>
  <p>However, there are some conditions under which it works.</p>
  <ul>
  <li>It works if I run the script via gluesparksubmit.</li>
  <li>It works if I use lambda expression instead of function declaration.</li>
  <li>It works if I declare a function within REPL and pass it as argument.</li>
  <li>It does not work if I put both <code>def func(): ()</code> and <code>.foreachPartition(func)</code> call in the same script.</li>
  <li>Moving the function declaration to another module also seems to work. But this couldn't be an option for I need to pack things in one job script.</li>
  </ul>
  <p>Could anyone please help me understand:</p>
  <ul>
  <li>why the error is thrown</li>
  <li>why the error is NOT thrown in other cases</li>
  </ul>
  <p>Complete error log: <a href="https://justpaste.it/37tj6" rel="nofollow noreferrer">https://justpaste.it/37tj6</a></p>