title: Py4JJavaError when running write_parquet in pyspark program
body: <p>When I run a pyspark program, I'm getting the following error:</p>
  <pre><code>Py4JJavaError: An error occurred while calling o148.parquet.
  : org.apache.spark.SparkException: Job aborted.
  </code></pre>
  <p>The particular line of code that causes the error is:</p>
  <pre><code># Send to .parquet
  write_parquet(df, config.source_path)
  </code></pre>
  <p>I've been reading online, and I think it's got to do with the different versions of installations. The relative packages in my Conda env are:</p>
  <pre><code>findspark 2.0.1
  mlflow 1.27
  pip 21.2.4
  py4j 0.10.9
  pylib 1.0.11
  pyspark 3.1.2 
  python 3.7.13
  boto3 1.24.19
  </code></pre>
  <p>and the java version is &quot;1.8.0_321&quot;
  Any suggestions on what the issue could be?</p>