I'm trying to do the following periodically (lets say once a week):

* download a couple of public datasets
* merge them together, resulting in a dictionary (I'm using Python) of ~2.5m entries
* upload/synchronize the result to Cloud Datastore so that I have it as "reference data" for other things running in the project

Synchronization can mean that some entries are updated, others are deleted (if they were removed from the public datasets) or new entries are created.

I've put together a python script using [google-cloud-datastore](https://pypi.org/project/google-cloud-datastore/) however the performance is abysmal - it takes around 10 hours (!) to do this. What I'm doing:

* iterate over the entries from the datastore
* look them up in my dictionary and decide if the need update / delete (if no longer present in the dictionary)
* write them back / delete them as needed
* insert any new elements from the dictionary

I already batch the requests (using .put\_multi, .delete\_multi, etc).

Some things I considered:

* Use [DataFlow](https://cloud.google.com/dataflow/). The problem is that each tasks would have to load the dataset (my "dictionary") into memory which is time and memory consuming
* Use the [managed import / export](https://cloud.google.com/datastore/docs/export-import-entities). Problem is that it produces / consumes some undocumented binary format (I would guess entities serialized as protocol buffers?)
* Use multiple threads locally to mitigate the latency. Problem is the google-cloud-datastore library has limited support for cursors (it doesn't have an "advance cursor by X" method for example) so I don't have a way to efficiently divide up the entities from the DataStore into chunks which could be processed by different threads

How could I improve the performance?