title: Pandas dataframe to parquet buffer in memory
body: <p>The use case is the following:</p>
  
  <ol>
  <li>Read data from external database and load it into pandas dataframe</li>
  <li>Transform that dataframe into parquet format buffer</li>
  <li>Upload that buffer to s3</li>
  </ol>
  
  <p>I've been trying to do step two in-memory (without having to store the file to disk in order to get the parquet format), but all the libraries I've seen so far, they always write to disk.</p>
  
  <p>So I have the following questions:</p>
  
  <ul>
  <li>Wouldn't it be more performant if the conversion was done in-memory since you don't have to deal with I/O disk overhead?</li>
  <li>As you increase the concurrent processes converting files and storing them into disk, couldn't we have issues regarding disk such as running out of space at some points or reaching throughput limit of the disk ?</li>
  </ul>