title: Reading in large files from S3 , and handling memory constraints
body: <p>The snippet of code here reads in a file that eventually parses and passes data into an api request. But some of the files may be very large - any tips how I can handle memory issues ?</p>
  <pre><code>addresses =    glob_common_vars.s3_resource.Object(glob_common_vars.s3_bucket_name,obj['Key'])
             with gzip.GzipFile(fileobj=addresses.get()[&quot;Body&quot;]) as gzipfile:
                 addresses = ndjson.loads(gzipfile.read().decode('utf-8')) ## may cause memory    issue
              
  </code></pre>