title: How can I reduce Google App Engine datastore latency?
body: <p>Through appstats, I can see that my datastore queries are taking about 125ms (api and cpu combined), but often there are long latencies (e.g. upto 12000ms) before the queries are executed. </p>
  
  <p>I can see that my latency from the datastore is not related to my query (e.g. the same query/data has vastly different latencies), so I'm assuming that it's a scheduling issue with app engine.  </p>
  
  <p>Are other people seeing this same problem ?  </p>
  
  <p>Is there someway to reduce the latency (e.g. admin console setting) ?</p>
  
  <p>Here's a screen shot from appstats. This servlet has very little cpu processing. It does a getObjectByID and then does a datastore query. The query has an OR operator so it's being converted into 3 queries by app engine. </p>
  
  <p><img src="https://i.stack.imgur.com/e2jwx.png" alt="appstats screenshot">
  .
  As you can see, it takes 6000ms before the first getObjectByID is even executed. There is no processing before the get operation (other than getting pm). I thought this 6000ms latency might be due to an instance warm-up, so I had increased my idle instances to 2 to prevent any warm-ups. </p>
  
  <p>Then there's a second latency around a 1000ms between the getObjectByID and the query. There's zero lines of code between the get and the query. The code simply takes the result of the getObjectByID and uses the data as part of the query. </p>
  
  <p>The grand total is 8097ms, yet my datastore operations (and 99.99% of the servlet) are only 514ms (45ms api), though the numbers change every time I run the servlet.  Here is another appstats screenshot that was run on the same servlet against the same data. 
  <img src="https://i.stack.imgur.com/jyVxS.png" alt="enter image description here"></p>
  
  <p>Here is the basics of my java code. I had to remove some of the details for security purposes.</p>
  
  <pre><code>user = pm.getObjectById(User.class, userKey);           
  //build queryBuilder.append(...
  final Query query = pm.newQuery(UserAccount.class,queryBuilder.toString());
  query.setOrdering("rating descending");
  query.executeWithArray(args); 
  </code></pre>
  
  <p>Edited:
  Using Pingdom, I can see that GAE latency varies from 450ms to 7,399ms, or 1,644% difference !! This is with two idle instances and no users on the site. 
  <img src="https://i.stack.imgur.com/xEHcC.png" alt="enter image description here"></p>