GAE datastore restore stops with The API call urlfetch.Fetch() took too long to respond and was cancelled
Question: I am following this guide
<https://cloud.google.com/appengine/docs/python/console/datastore-backing-up-
restoring#restoring_data_to_another_app> on how to backup data in one GAE app
and restore it in another.

But every time I restore the backup on the target application I get the error:

    
    
    The API call urlfetch.Fetch() took too long to respond and was cancelled.
    

Any ideas what I am doing wrong?


Answer: Your urlfetch.Fetch() is taking too long (greater than 60 seconds) to respsond
and so it is timing out. Here is an article about it
<https://cloud.google.com/appengine/articles/deadlineexceedederrors>

One solution is to use task queues. Task queues have a longer timeout or, more
appropriately, let you chop the job up into smaller parts.
<https://cloud.google.com/appengine/docs/python/taskqueue/>

Here is a simple example of how to do this with "push" task queues. I realize
going from one datastore model to another might not be the redundancy you are
looking for. You may want to backup the datastore entities to another app
entirely or another type of database or cloud service. You also probably have
multiple models you are backing up. This is just a simple example of setting
up and schedule a "push" task queue using a cron job every 24 hours:

first you have to add "deferred" to the builtins in your app.yaml:

    
    
    builtins:
    - deferred: on
    

Next you need to create a second datastore model we will call "Backup" just
copy paste your old model and rename it Backup - It helps to use an identical
version of the same model for backups rather than the same model because you
can give them the same the primary and the backup the same key:

    
    
    class Backup(db.Model): # example
        prop1 = db.StringProperty()
        prop2 = db.StringListProperty()
        prop3 = db.StringProperty()
    

Next setup a cron job in your cron.yaml:

    
    
    - description: Creates a backup of the target db every 24 hours at 10:45 GMT
    url: /backup
    schedule: everyday 10:45
    

Add /backup to your app.yaml handlers:

    
    
    - url: /backup
    script: mybackup.py
    login: admin
    

Finally, create mybackup.py

    
    
    from google.appengine.ext import deferred
    from google.appengine.ext import db
    #from google.appengine.ext import ndb
    
    def backup_my_model(model_name):
        """
        Takes all enities in the model_name model and copies it to Backup model
        """
        logging.info("Backing up %s" % model_name)
        query = db.GqlQuery('SELECT * From %s ' % model_name)
        for primary_db in query:
            backup = Backup(key_name = primary_db.key_name)
            backup.prop1 = primary_db.prop1
            backup.prop2 = primary_db.prop2
            ...
            backup.put()
    
    
    
    deferred.defer(backup_my_model, MyModel) #where MyModel is the model you want to backup
    deferred.defer(backup_my_model, MyOtherModel)
    ...
    deferred.defer(backup_my_model, MyFinalModel)
    

I hope that helps.

