title: Can you upload to S3 using a stream rather than a local file?
body: <p>I need to create a CSV and upload it to an S3 bucket. Since I'm creating the file on the fly, it would be better if I could write it directly to S3 bucket as it is being created rather than writing the whole file locally, and then uploading the file at the end.</p>
  
  <p>Is there a way to do this? My project is in Python and I'm fairly new to the language. Here is what I tried so far:</p>
  
  <pre><code>import csv
  import csv
  import io
  import boto
  from boto.s3.key import Key
  
  
  conn = boto.connect_s3()
  bucket = conn.get_bucket('dev-vs')
  k = Key(bucket)
  k.key = 'foo/foobar'
  
  fieldnames = ['first_name', 'last_name']
  writer = csv.DictWriter(io.StringIO(), fieldnames=fieldnames)
  k.set_contents_from_stream(writer.writeheader())
  </code></pre>
  
  <p>I received this error: BotoClientError: s3 does not support chunked transfer</p>
  
  <p><strong>UPDATE: I found a way to write directly to S3, but I can't find a way to clear the buffer without actually deleting the lines I already wrote. So, for example:</strong></p>
  
  <pre><code>conn = boto.connect_s3()
  bucket = conn.get_bucket('dev-vs')
  k = Key(bucket)
  k.key = 'foo/foobar'
  
  testDict = [{
      "fieldA": "8",
      "fieldB": None,
      "fieldC": "888888888888"},
      {
      "fieldA": "9",
      "fieldB": None,
      "fieldC": "99999999999"}]
  
  f = io.StringIO()
  fieldnames = ['fieldA', 'fieldB', 'fieldC']
  writer = csv.DictWriter(f, fieldnames=fieldnames)
  writer.writeheader()
  k.set_contents_from_string(f.getvalue())
  
  for row in testDict:
      writer.writerow(row)
      k.set_contents_from_string(f.getvalue())
  
  f.close()
  </code></pre>
  
  <p>Writes 3 lines to the file, however I'm unable to release memory to write a big file. If I add:</p>
  
  <pre><code>f.seek(0)
  f.truncate(0)
  </code></pre>
  
  <p>to the loop, then only the last line of the file is written. Is there any way to release resources without deleting lines from the file?</p>