title: Synchronising data from MySql to DynamoDb
body: <p>I'm currently trying to find a (good) solution on how to synchronise data from an external MySql database which is completely separated from anything AWS into an AWS DynamoDb.</p>
  
  <p>The sync. process should run every day at around 12:00PM and should grab the latest inserted item from DynamoDb containing a create date to make sure we only get MySql data from that given date/time when the sync. runs. The sync. would typically transfer around 110.000 records every day.</p>
  
  <p>One thing to note: we're using .NET where I work.</p>
  
  <p>From what I've understood, there are a few AWS services that can help me do this:</p>
  
  <p><strong>ERM</strong> (<a href="https://aws.amazon.com/emr/?nc2=h_m1" rel="nofollow noreferrer">Link</a>)</p>
  
  <p>AWS ERM seems like the way to go, but it seems that Hive scripts are not able to communicate with external MySql databases? Or am I wrong here? I find it hard to find any usable Hive script examples.</p>
  
  <p><strong>Data Pipeline</strong> (<a href="https://aws.amazon.com/datapipeline/?nc2=h_m1" rel="nofollow noreferrer">Data Pipeline</a>)</p>
  
  <p>From what I've understood, Data Pipeline is best used when your db schema is the exact same in both ends, which is not the case here, since we're reading from a MySql database into a DynamoDb. The structure is not completely 1:1.</p>
  
  <p>Third option would be to create a Windows Service which runs a piece of C# code to read data from MySql and store it in DynamoDb. Only thing I'm worried about here, is performance :-) Looping through 100.000+ records processing them and then store each in DynamoDb doesn't seem appealing to me.</p>
  
  <p>Does anyone have any experience with this they'd like to share? :-) Concrete examples would be very welcome. Also, if I've missed any kind of service/other way of implementing this, please let me know.</p>