title: Hitting AWS Lambda Memory limit in Python
body: <p>I am looking for some advice on this project.  My thought was to use Python and a Lambda to aggregate the data and respond to the website.  The main parameters are date ranges and can be dynamic.</p>
  
  <p><strong>Project Requirements:</strong></p>
  
  <ul>
  <li>Read data from monthly return files stored in JSON (each file contains roughly 3000 securities and is 1.6 MB in size)</li>
  <li>Aggregate the data into various buckets displaying counts and returns for each bucket (for our purposes here lets say the buckets are Sectors and Market Cap ranges which can vary)</li>
  <li>Display aggregated data on a website</li>
  </ul>
  
  <p><strong>Issue I face</strong>
  I have successfully implemted this in an AWS Lambda, however in testing requests that are 20 years of data (and yes I get them), I begin to hit the memory limits in AWS Lambda.</p>
  
  <p><strong>Process I used:</strong>
  All files are stored in S3, so I use the boto3 library to obtain the files, reading them into memory.  This is still small and not of any real significance.</p>
  
  <p>I use <code>json.loads</code> to convert the files into a pandas dataframe.  I was loading all of the files into one large dataframe. - This is where the it runs out of memory.</p>
  
  <p>I then pass the dataframe to custom aggregations using <code>groupby</code> to get my results.  This part is not as fast as I would like but does the job of getting what I need.</p>
  
  <p>The end result dataframe that is then converted back into JSON and is less than 500 MB.</p>
  
  <p>This entire process when it works locally outside the lambda is about 40 seconds.<br>
  I have tried running this with threads and processing single frames at once but the performance degrades to about 1 min 30 seconds.  </p>
  
  <p>While I would rather not scrap everything and start over, I am willing to do so if there is a more efficient way to handle this.  The old process did everything inside of node.js without the use of a lambda and took almost 3 minutes to generate.</p>
  
  <p><strong>Code currently used</strong>
  I had to clean this a little to pull out some items but here is the code used.
  Read data from S3 into JSON this will result in a list of string data.</p>
  
  <pre><code> while not q.empty():
              fkey = q.get()
  
  
              try:
                  obj = self.s3.Object(bucket_name=bucket,key=fkey[1])
                  json_data = obj.get()['Body'].read().decode('utf-8')
  
                  results[fkey[1]] = json_data
              except Exception as e:
                  results[fkey[1]] = str(e)
              q.task_done()
  </code></pre>
  
  <p>Loop through the JSON files to build a dataframe for working</p>
  
  <pre><code>for k,v in s3Data.items():        
          lstdf.append(buildDataframefromJson(k,v))
  
  def buildDataframefromJson(key, json_data):
  
      tmpdf = pd.DataFrame(columns=['ticker','totalReturn','isExcluded','marketCapStartUsd',
                                    'category','marketCapBand','peGreaterThanMarket', 'Month','epsUsd']
                           )
       #Read the json into a dataframe
      tmpdf = pd.read_json(json_data,
                           dtype={
                              'ticker':str,
                              'totalReturn':np.float32,
                              'isExcluded':np.bool,
                              'marketCapStartUsd':np.float32,
                              'category':str,
                              'marketCapBand':str,
                              'peGreaterThanMarket':np.bool,
                              'epsUsd':np.float32
                              })[['ticker','totalReturn','isExcluded','marketCapStartUsd','category',
                          'marketCapBand','peGreaterThanMarket','epsUsd']]
      dtTmp = datetime.strptime(key.split('/')[3], "%m-%Y")
      dtTmp = datetime.strptime(str(dtTmp.year) + '-'+ str(dtTmp.month),'%Y-%m')
      tmpdf.insert(0,'Month',dtTmp, allow_duplicates=True)
  
      return tmpdf
  </code></pre>