title: Creating a data lake from a DynamoDB table
body: <p>We have a service where a DynamoDB table ~50GB is our feature repository, which we use for real-time, online applications.</p>
  <p>We want to create a data lake from this table for historical data, model training and analytics insights. We want to guarantee a 30-minutes &quot;freshness&quot; of data lake data w.r.t. the original table.</p>
  <p>However, I'm confused on what could be a good architecture for this: my understanding of data lakes is that you should use a storage service (i.e., S3) to store the <strong>raw data</strong> with no processing. Then, you perform ETL jobs, where you transform, process and filter the data (e.g., using Glue) before using for whatever app.</p>
  <p><strong>But here is my doubt:</strong> does this means that we have to dump the DynamoDB table into S3 every 30 minutes? This can be <a href="https://aws.amazon.com/blogs/aws/new-export-amazon-dynamodb-table-data-to-data-lake-amazon-s3/" rel="nofollow noreferrer">easily done</a>, but it sounds weird (this would result in ~876TB/year).</p>
  <p>Am I missing something in the data lake pipeline?</p>