title: Is saving a HUGE dask dataframe into parquet possible?
body: <p>I have a dataframe made up of 100,000+ rows and each row has 100,000 columns, totally to 10,000,000,000 float values. </p>
  
  <p>I've managed to read them in previously in a <code>csv</code> (tab-separated) file and I successfully read them to a 50 cores Xeon machine with 250GB RAM and try to write it out as a <code>.parq</code> directory as such:</p>
  
  <p>The floats in <code>huge.csv</code> were saved as strings and it is 125GB. </p>
  
  <pre><code>import dask.dataframe as dd
  filename = 'huge.csv'
  df = dd.read_csv(filename, delimiter='\t', sample=500000000)
  df.to_parquet('huge.parq')
  </code></pre>
  
  <p>It has been writing to the <code>huge.parq</code> for close to a week and the directory is 14GB and it seems like the process of saving <code>.to_parquet</code> is not going to stop any time soon. </p>
  
  <p>And <code>free -mh</code> is showing that there's still memory left available but the time it's taking to save the <code>.parq</code> directory is tremendously slow:</p>
  
  <pre><code>$ free -mh
                total        used        free      shared  buff/cache   available
  Mem:           251G         98G         52G         10M        101G        152G
  Swap:          238G          0B        238G
  </code></pre>
  
  <p>The questions are:</p>
  
  <ul>
  <li><p>Given the size of the dataframe and machine, is it feasible to save the dask dataframe to a parquet file at all? </p></li>
  <li><p>Is it normal for <code>dask</code> and <code>fastparquet</code> to take so long to save huge dataframes?</p></li>
  <li><p>Is there some way to estimate the time it will take to save a parquet file?</p></li>
  </ul>