Import large chunk of data into Google App Engine Data Store at one go
Question: I have a large CSV file, approx 10 MB in size, which contains all the data
which need to be imported in the Google App Engine DataStore. I tried
following approaches to perform import but all the times it failed in half
way.

  * Import using mapping a command to url and then executing url, failed because of request time out...
  * Import using creating cron job, but got DeadlineExceededError...
  * Import using remort_api_shell, but got Operation timed out.

Can you please suggest me and approch (using dummy data you can imagine) how
to do it... Suggestion with code will be more helpful..

** I am using Python and google's web app framework to develop the said app.


Answer: you can post row by row. using built in bulk loader.

<http://code.google.com/appengine/docs/python/tools/uploadingdata.html>

this is good article.

and here is my contactloader.py that i used 2 years ago for reference. it is
more sophisticated since last i used but still.....

    
    
    import datetime
    from google.appengine.ext import db
    from google.appengine.tools import bulkloader
    
    class Contact(db.Model):
    
        date = db.DateTimeProperty(auto_now_add=True)
    
        owner = db.StringProperty()
    
        companyname = db.StringProperty()
    
        companyemail = db.EmailProperty()
    
    def myfunc(x):
        temp = x.split(":mailto:")
        if len(temp) > 0:
            temp = temp[-1].split(":")
        else:
            return "defaultvalue"
        if len(temp) > 0:
            temp = temp[0]
        else:
            return "defaultvalue"
        temp = temp.split("<1>")[0]
        if temp is None or len(temp) < 5:
            return "defaultvalue"
        return temp
    
    def mysecfunc(x):
        return x.split("<0>")[0]
    
    class ContactLoader(bulkloader.Loader):
        def __init__(self):
            bulkloader.Loader.__init__(self, 'Contact',
                                       [
                                        ('companyname',mysecfunc),
                                        ('owner', lambda x:"somevalue"),
                                        ('companyemail',myfunc),
                                        ("date",lambda x:datetime.datetime.now()),
                                       ])
    
    loaders = [ContactLoader]
    

