title: Snowflake python connector not working on larger data set in AWS Lambda
body: <p>I'm using Snowflakes python connector to try to retrieve a set of data from our data warehouse for processing. This job is executing within a AWS lambda function and has trouble when the rows being returning back is ~20 or so. When I set a <code>limit 10</code> or <code>limit 20</code> I'm able to get the data set back. If I leave <code>limit</code> off then it struggles trying to get the result set of only 65 rows. </p>
  
  <p>The memory and timeout values in my lambda are already at the max and the data set exported to CSV was only 300KB. Running locally this query returns just fine so it may have something to do with memory size but the data being returning is not that large really. </p>
  
  <pre><code>connector = snowflake.connector.connect(
      account=os.environ['SNOWFLAKE_ACCOUNT'],
      user=os.environ['SNOWFLAKE_USER'],
      password=os.environ['SNOWFLAKE_PASSWORD'],
      role="MY_ROLE",
      ocsp_response_cache_filename="/tmp/.cache/snowflake/"
                                   "ocsp_response_cache",
  )
  print("Connected to snowflake")
  cursor = connector.cursor(DictCursor)
  cursor.execute('USE DATA.INFORMATION_SCHEMA')
  
  query = "SELECT * FROM TABLE WHERE X=Y"  # FAKE QUERY
  
  print("Execute query: \n\t{0}".format(query))
  cursor.execute(query)
  print("Execute query done!")
  posts = []
  processed = 0
  for rec in cursor:
      processed += 1
      print("Processed count: {}".format(processed))
      posts.append(rec)
  
  # These attempts also didn't work. 
  # posts = cursor.fetchmany(size=cursor.rowcount)
  # posts = cursor.fetchall()
  
  cursor.close()
  </code></pre>
  
  <p><code>processed</code> integer value gets up to 17 records but then halts. My logs are outputting a lot of stuff about chunks not being ready to consume and eventually the lambda just times out</p>
  
  <pre><code>[1531919679073] [DEBUG] 2018-07-18T13:14:39.72Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 Chunk Downloader in memory
  [1531919679073] Execute query done!
  [1531919679073] [DEBUG] 2018-07-18T13:14:39.73Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 chunk index: 0, chunk_count: 2
  [1531919679073] [DEBUG] 2018-07-18T13:14:39.73Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 next_chunk_to_consume=1, next_chunk_to_download=3, total_chunks=2
  [1531919679073] [DEBUG] 2018-07-18T13:14:39.73Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 waiting for chunk 1/2 in 1/10 download attempt
  [1531919679073] [DEBUG] 2018-07-18T13:14:39.73Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 chunk 1/2 is NOT ready to consume in 10/3600(s)
  [1531919679073] [DEBUG] 2018-07-18T13:14:39.73Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 downloading chunk 1/2
  [1531919679074] [DEBUG] 2018-07-18T13:14:39.73Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 use chunk headers from result
  [1531919679074] [DEBUG] 2018-07-18T13:14:39.74Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 started getting the result set 1: https://sfc-va-ds1-customer-stage.s3.amazo
  naws.com/fwoi-s-vass0007/results/7b9cf772-a061-47ab-8e9f-43dbfcd923c9_0/main/data_0_0_0?x-amz-server-side-encryption-customer-algorithm=AES256&amp;response-content-e
  ncoding=gzip&amp;AWSAccessKeyId=AKIAJKHCJ73YL7MD6ZRA&amp;Expires=1531941279&amp;Signature=VvGOkLNvE%2FHVMaUXoeQMn6cFUOY%3D
  [1531919679074] [DEBUG] 2018-07-18T13:14:39.74Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 Active requests sessions: 1, idle: 0
  [1531919679074] [DEBUG] 2018-07-18T13:14:39.74Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 remaining request timeout: 3600, retry cnt: 1
  [1531919679074] [DEBUG] 2018-07-18T13:14:39.74Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 socket timeout: 60
  [1531919679075] [INFO] 2018-07-18T13:14:39.75Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 Starting new HTTPS connection (1): sfc-va-ds1-customer-stage.s3.amazonaws.com
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.75Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 downloading chunk 2/2
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.76Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 use chunk headers from result
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.76Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 started getting the result set 2: https://sfc-va-ds1-customer-stage.s3.amazo
  naws.com/fwoi-s-vass0007/results/7b9cf772-a061-47ab-8e9f-43dbfcd923c9_0/main/data_0_0_1?x-amz-server-side-encryption-customer-algorithm=AES256&amp;response-content-e
  ncoding=gzip&amp;AWSAccessKeyId=AKIAJKHCJ73YL7MD6ZRA&amp;Expires=1531941279&amp;Signature=F5ix8FcsLO1dM8sWsZXZYx4uHM8%3D
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.76Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 Converted retries value: 1 -&gt; Retry(total=1, connect=None, read=None, redire
  ct=None)
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.76Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 Converted retries value: 1 -&gt; Retry(total=1, connect=None, read=None, redire
  ct=None)
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.76Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 Active requests sessions: 2, idle: 0
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.76Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 remaining request timeout: 3600, retry cnt: 1
  [1531919679078] [DEBUG] 2018-07-18T13:14:39.76Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 socket timeout: 60
  [1531919679078] [INFO] 2018-07-18T13:14:39.77Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 Starting new HTTPS connection (1): sfc-va-ds1-customer-stage.s3.amazonaws.com
  [1531919681581] [DEBUG] 2018-07-18T13:14:41.580Z 26284dc8-8a8c-11e8-95ac-3ff42bd28642 chunk 1/2 is NOT ready to consume in 160/3600(s)
  [1531919689074] [DEBUG] 2018-07-18T13:14:49.73Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 chunk 1/2 is NOT ready to consume in 20/3600(s)
  [1531919691581] [DEBUG] 2018-07-18T13:14:51.581Z 26284dc8-8a8c-11e8-95ac-3ff42bd28642 chunk 1/2 is NOT ready to consume in 170/3600(s)
  [1531919699074] [DEBUG] 2018-07-18T13:14:59.74Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 chunk 1/2 is NOT ready to consume in 30/3600(s)
  [1531919701581] [DEBUG] 2018-07-18T13:15:01.581Z 26284dc8-8a8c-11e8-95ac-3ff42bd28642 chunk 1/2 is NOT ready to consume in 180/3600(s)
  [1531919709074] [DEBUG] 2018-07-18T13:15:09.74Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 chunk 1/2 is NOT ready to consume in 40/3600(s)
  [1531919711582] [DEBUG] 2018-07-18T13:15:11.581Z 26284dc8-8a8c-11e8-95ac-3ff42bd28642 chunk 1/2 is NOT ready to consume in 190/3600(s)
  [1531919712739] [DEBUG] 2018-07-18T13:15:12.738Z 26284dc8-8a8c-11e8-95ac-3ff42bd28642 Incremented Retry for (url='/fwoi-s-vass0007/results/7b9cf772-a061-47ab-8e9
  f-43dbfcd923c9_0/main/data_0_0_0?x-amz-server-side-encryption-customer-algorithm=AES256&amp;response-content-encoding=gzip&amp;AWSAccessKeyId=AKIAJKHCJ73YL7MD6ZRA&amp;Expire
  s=1531941131&amp;Signature=mW6nXerwYHhnfwfPdRF0So1tpIQ%3D'): Retry(total=0, connect=None, read=None, redirect=None)
  [1531919719075] [DEBUG] 2018-07-18T13:15:19.75Z 7e3420c6-8a8c-11e8-a97e-c53a2c591430 chunk 1/2 is NOT ready to consume in 50/3600(s)
  </code></pre>