title: What is the fastest way to save a large pandas DataFrame to S3?
body: <p>I am trying to figure out what is the fastest way to write a LARGE pandas DataFrame to S3 filesystem. I am currently trying two ways:</p>
  
  <p>1) Through gzip compression (BytesIO) and boto3</p>
  
  <pre class="lang-py prettyprint-override"><code>gz_buffer = BytesIO()
  
  with gzip.GzipFile(mode='w', fileobj=gz_buffer) as gz_file:
      df.to_csv(TextIOWrapper(gz_file, 'utf8'), index=False)
  
  s3_resource = boto3.resource('s3')
  s3_object = s3_resource.Object(bucket, s3_path + name_zip)
  s3_object.put(Body=gz_buffer.getvalue())
  </code></pre>
  
  <p>which for a dataframe of 7M rows takes around 420seconds to write to S3.</p>
  
  <p>2) Through writing to csv file without compression (StringIO buffer)</p>
  
  <pre class="lang-py prettyprint-override"><code>csv_buffer = StringIO()
  data.to_csv(csv_buffer)
  s3_resource = boto3.resource('s3')
  s3_resource.Object(bucket, s3_path + name_csv).put(Body=csv_buffer.getvalue())
  </code></pre>
  
  <p>which takes around 371 seconds...</p>
  
  <p>The question is:
  Is there any other faster way to write a pandas dataframe to S3?</p>