title: Pyspark partition data by a column and write parquet
body: <p>I need to write parquet files in seperate s3 keys by values in a column. The column <code>city</code> has thousands of values. Iteration using for loop, filtering dataframe by each column value and then writing parquet is very slow. Is there any way to partition the dataframe by the column <code>city</code> and write the parquet files?</p>
  <p>What I am currently doing -</p>
  <pre><code>for city in cities:
    print(city)
    spark_df.filter(spark_df.city == city).write.mode('overwrite').parquet(f'reporting/date={date_string}/city={city}')
  </code></pre>