set size of Parquet output files in dask
The current behaviour is by design, allowing each worker to process a partition independently, and write to files which no other process is writing to. Otherwise, there would need to be some kind of lock, or some consolidation step after writing for each directory.
What you could do, is to use set_index to shuffle the data into one partition for each value of the column you want to partition by (perhaps using the divisions= keyword); now a to_parquet would result in a file for each of these values. If you wanted the files to automatically end up in the correct directories, and have the now-redundant index trimmed, you would want to use to_delayed() and create a delayed function, which takes one partition (a pandas dataframe) and writes it to the correct location.
when using the dask dataframe to_parquet method is there any way to set the default parquet file size like in spark ?
my problem is that when I save it with the partition_on kwarg i get several small files per partition dir and thus resulting very slow queries using "Amazon Athena".
The intermediate desired result (if file size control is not available) is n files (right now 1 will suffice) per partition dir.
The only way i thought of guaranteeing 1 file per partition dir is repartitioning to one partition and then using the to_parquet method (however this is highly inefficient).
is there a better way ?