title: The right way to make DynamoDB data searchable in Athena?
body: <p>I need to have, a slowly changing, AWS DynamoDb periodically dumped on S3, for querying it on Athena. It needs to be ensured that data available to Athena is not much behind what's available on DynamoDb (maximum lag of 1 hour)</p>
  <p>I am aware of the following two approaches:</p>
  <ol>
  <li><p>Use EMR (from Data Pipeline) to <a href="https://github.com/aws-samples/data-pipeline-samples/tree/master/samples/DynamoDBExport" rel="noreferrer">export</a> the entire DynamoDb</p>
  <p>Advantage of this approach is that with a single EMR script (run hourly), compressed Parquet files, which are directly searchable on Athena, can be dumped on S3. However, a big disadvantage of this approach is that while only a small number of records change in an hour, the entire dump needs to be taken, requiring significantly higher read capacity in DynamoDb and higher EMR resources.</p>
  </li>
  <li><p>Use <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.html" rel="noreferrer">DynamoDB Streams</a> to reflect any changes in DynamoDb on S3.</p>
  <p>This has the advantage of not needing to process unchanged data on DynamoDb, thus reduces the need of significantly higher read capacity than whats needed in normal operations. However, a follow up script (probably another EMR job) would be needed to consolidate the per record files generated by DynamoDb streams, else performance of Athena gets severely impacted because of large number of files.</p>
  </li>
  </ol>
  <p>Are there any other approaches which can do better than these?</p>