I'm relatively new to AWS and am working through strategies that best support a specific business requirement of the service we're developing.

Among our challenges:

1. We need to pull in a very large data set (hundreds of thousands of records) from a third-party API, which delivers paginated records in max groups of 50;
2. We need to assign two unique, internal keys to each of the imported records;
3. We need to update the imported records by making regularly scheduled calls for updated and new records; and
4. In time, we will be adding records from additional sources - and will need to reconcile (match, duplicates) data from multiple sources.

Our goal is to keep this data separate from our main database, so that we can independently manage and query it as needed. As such, the strategy we've been considering is:

1. Making the API calls in a recursive lambda function (due to pagination);
2. Storing the results of the call in an S3 bucket as a single - or multiple - json files;
3. Pulling the S3 data into a non-relational DB.

We're having some problems here though:

* Given that the initial import will take several hours, our lambda is timing out at 15 min (hard limit);
* What is the best way to assign our own unique keys to the incoming data (one key should ideally be generated by taking incoming data and reformatting it to our needs); and
* What is the best strategy to update these records with updated information from the source or a third party?

Not sure if this should be broken out into three questions - but this seems to be the best way to provide the full context.