Add lots of data to Google App Engine NDB (Python)
Question: I'm working on a project that requires me to import lots of data (on the scale
of tens of thousands of entities) into the Google App Engine NDB.

The data is stored in a text file, for which I wrote a parsing program that
generates a list of the entities found in the file, which I then write into
the database using the `put_multi()` method.

When I apply this to testing data sets of say a couple hundred up to about a
thousand entries, it works fine; applying it to the real data set (at about
30,000 entries right now, but it will grow), however, throws a
`DeadlineExceededError`. I'm guessing that means the program is running too
long and App Engine cuts it off.

So my question is, how do I run code that takes a long time to complete while
it iterates over a bunch of data?

I've read something about a task queue or something, but that seems to be more
aimed at a short task that needs to be repeated a bunch of times (while, here,
I'm really just waiting for the `put_multi()` to complete).

It's also a pretty bad UX having the page take this long to load. Is there any
way for me to have my code that runs when the user initializes the data import
just sort of kickstart the actual parsing and writing code behind the scenes
on the server while the user gets a page back that says the data is being
processed in the background?

Thanks


Answer: there is a put_multi_async(), which will take like 10 ms to complete and then
do the put in the background. You're still limited by the datastore deadline,
which :

> The default deadline is 60 seconds
> (<https://developers.google.com/appengine/docs/java/datastore/queries#Java_Data_consistency>)

you might need to spread your async puts in different calls (maybe a couple of
thousand each, write speed will depend on the exact size of your entity)

EDIT : as per mgilson's comment,

> > You're also limited by memory restrictions. If you have enough memory to
> pull in the entire file, then task-queues might be the answer (put a bunch
> of different pieces of the file into different tasks)

