title: How to stream a large gzipped .tsv file from s3, process it, and write back to a new file on s3?
body: <p>I have a large file <code>s3://my-bucket/in.tsv.gz</code> that I would like to load and process, write back its processed version to an s3 output file <code>s3://my-bucket/out.tsv.gz</code>.</p>
  <ol>
  <li>How do I streamline the <code>in.tsv.gz</code> directly from s3 without loading all the file to memory (it cannot fit the memory)</li>
  <li>How do I write the processed gzipped stream directly to s3?</li>
  </ol>
  <p>In the following code, I show how I was thinking to load the input gzipped dataframe from s3, and how I would write the <code>.tsv</code> if it were located locally <code>bucket_dir_local = ./</code>.</p>
  <pre><code>import pandas as pd
  import s3fs
  import os
  import gzip
  import csv
  import io
  
  bucket_dir = 's3://my-bucket/annotations/'
  df = pd.read_csv(os.path.join(bucket_dir, 'in.tsv.gz'), sep='\t', compression=&quot;gzip&quot;)
  
  bucket_dir_local='./'
  # not sure how to do it with an s3 path
  with gzip.open(os.path.join(bucket_dir_local, 'out.tsv.gz'), &quot;w&quot;) as f:
      with io.TextIOWrapper(f, encoding='utf-8') as wrapper:
          w = csv.DictWriter(wrapper, fieldnames=['test', 'testing'], extrasaction=&quot;ignore&quot;)
          w.writeheader()
          for index, row in df.iterrows():
              my_dict = {&quot;test&quot;: index, &quot;testing&quot;: row[6]}
              w.writerow(my_dict)
  </code></pre>
  <p><strong>Edit</strong>: <a href="https://github.com/RaRe-Technologies/smart_open" rel="nofollow noreferrer">smart_open</a> looks like the way to go.</p>