Why do Parquet files generate multiple parts in Pyspark?
It's not just for parquet but rather a spark feature where to avoid network io it writes each shuffle partition as a 'part...' file on disk and each file as you said will have compression and efficient encoding by default.
So Yes it is directly related to parallel processing
After some extensive research I have figured that
Parquet is a column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk.
However, I am unable to understand why parquet writes multiple files when I run df.write.parquet("/tmp/output/my_parquet.parquet") despite supporting flexible compression options and efficient encoding.
Is this directly related to parallel processing or similar concepts?