How do I reduce the time to write large CSV (12 Go) to GCS bucket with Pyspark? (>6h)
The read will be a single Spark task, but the transformation and write will be parallelised by Spark, so increasing the size of your cluster will increase the parallelism and correspondingly decrease the execution time.
I have a Dataproc cluster to transform many CSV files stored in GCS, but when I write the CSV to a GCS bucket this takes too much time, 7h for one file (12 go).
This is the code that I used to read & write the CSV:
df = spark.read.options(delimiter='¤',header="true").csv('gs://'+bucket_name+'/'+file_name)
df.write.format('csv').option("header","True").option('sep','¤').save('gs://'+bucket_name+'/'+target_obj_name+'_tmp'+'/'+filename)
How can i optimise this operation?