title: Stream huge zip files on S3 using Lambda and boto3
body: <p>I have a bunch of CSV files compressed as one zip on S3. I only need to process one CSV file inside the zip using AWS lambda function</p>
  <pre><code>import boto3
  from zipfile import ZipFile
  
  BUCKET = 'my-bucket'
  s3_rsc = boto3.resource('s3')
  
  def zip_stream(zip_f='app.zip', bkt=BUCKET, rsc=s3_rsc):
      obj = rsc.Object(
          bucket_name=bkt,
          key=zip_f
      )
  
      return ZipFile(BytesIO(obj.get()['Body'].read()))
  
  
  zip_obj = zip_stream()
  csv_dat = zip_obj.read('one.csv')
  </code></pre>
  <p>The above snippet works well with test zip files, however, it fails with <strong>memory error</strong> if the zip file size exceeds 0.5G.</p>
  <p>Error Message</p>
  <blockquote>
  <p>{   &quot;errorMessage&quot;: &quot;&quot;,   &quot;errorType&quot;: &quot;MemoryError&quot;,   &quot;stackTrace&quot;:
  [
  &quot;  File &quot;/var/task/lambda_function.py&quot;, line 12, in handler\n    all_files = files_in_zip()\n&quot;,
  &quot;  File &quot;/var/task/lambda_function.py&quot;, line 36, in files_in_zip\n    zippo = zip_stream()\n&quot;,
  &quot;  File &quot;/var/task/lambda_function.py&quot;, line 32, in zip_stream\n    return ZipFile(BytesIO(obj.get()['Body'].read()))\n&quot;,
  &quot;  File &quot;/var/runtime/botocore/response.py&quot;, line 77, in read\n    chunk = self._raw_stream.read(amt)\n&quot;,
  &quot;  File &quot;/var/runtime/urllib3/response.py&quot;, line 515, in read\n    data = self._fp.read() if not fp_closed else b&quot;&quot;\n&quot;,
  &quot;  File &quot;/var/lang/lib/python3.8/http/client.py&quot;, line 468, in read\n    s = self._safe_read(self.length)\n&quot;,
  &quot;  File &quot;/var/lang/lib/python3.8/http/client.py&quot;, line 609, in _safe_read\n    data = self.fp.read(amt)\n&quot;   ] }</p>
  </blockquote>
  <p>Is there an option to <code>stream</code>/lazyload the zipfile to mitigate memory issues?</p>
  <p><em>Note</em> - I also referred an old post(<a href="https://stackoverflow.com/questions/7624900/how-can-i-use-boto-to-stream-a-file-out-of-amazon-s3-to-rackspace-cloudfiles">How can I use boto to stream a file out of Amazon S3 to Rackspace Cloudfiles?</a>) which spoke about streaming a file but not zip</p>