title: Using AWS To Process Large Amounts Of Data With Serverless
body: <p>I have about 300,000 transactions for each user in my DynamoDB database.</p>
  
  <p>I would like to calculate the taxes based on those transactions in a serverless manner, if that is the cheapest way.</p>
  
  <p>My thought process was that I should use AWS Step Functions to grab all of the transactions, store them into Amazon S3, then use AWS Step Functions to iterate over each row in the CSV file. The problem is that once I read a row in the CSV, I would have to store it in memory so that I can use it for later calculations. If this Lambda function runs out of time, then I have no way to save the state, so this route is not plausible.</p>
  
  <p>Another route which would be expensive, is to have two copies of each transaction in DynamoDB and perform the operations on the copy Table, keeping the original data untouched. The problem with this is that the DynamoDB table is eventually consistent and there could be a scenario where I read a dirty item.</p>