title: How can write out my dataframe as one file per value of a given column?
body: <p>I have a dataframe with a column called <code>key</code> with values ranging from <code>00</code> to <code>FF</code> (as strings). Ideally I would write that dataframe as parquet as one or more files per value of <code>key</code>. That is, I don't want a partition to have more that one possible value for <code>key</code>.</p>
  <p>My current approach is not working, as it produces a single file:</p>
  <pre><code>import dask.dataframe as dd
  
  df = dd.read_csv(
          &quot;/Users/ecerulm/Downloads/test/**/*.txt.gz&quot;,
          include_path_column=True,
          sep=&quot;\t&quot;,
          compression='gzip',
          blocksize=None)
  df['basename'] = (df.path.str.rpartition('/')[2])
  df['MAC'] = df.basename.str.partition('.')[2]
  df['MAC'] = df.MAC.str.partition('.')[0]
  df['key'] = df.MAC.str[-2:]
  df = df.set_index('key')
  df = df.repartition(partition_size='100MB')
  df = df.shuffle(['key'])
  
  dd.to_parquet(df, './output.parquet')
  </code></pre>
  <p>Is there any way to ensure that the partitions are &quot;cleanly&quot; partitioned on a column (I thought that <a href="https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.Series.shuffle" rel="nofollow noreferrer">df.shuffle()</a> would do that but it seems does not guarantee that each partition will only contain a single value of the <code>on</code> columns).</p>