title: Datastore NDB best practices when querying and extracting thousands of rows
body: <p>I'm using the High Replication Datastore, along with <code>ndb</code>. I have a <code>kind</code> with over 27,000 entities, which isn't that much. Supposedly the datastore is efficient in querying and extracting large amounts of data, but whenever I query over that kind, queries take a long time to finish (I've even got DeadlineExceededErrors).</p>
  
  <p>I have a model where I store keywords and URLs I want to index in Google:</p>
  
  <pre><code>class Keywords(ndb.Model):
      keyword = ndb.StringProperty(indexed=True)
      url = ndb.StringProperty(indexed=True)
      number_articles = ndb.IntegerProperty(indexed=True)
      # Some other attributes... All attributes are indexed
  </code></pre>
  
  <p>My current use cases are to build my Sitemap, and to fetch my top 20 keywords to link from my hope page.</p>
  
  <p>When I fetch many entities, I usually do:</p>
  
  <pre><code>Keywords.query().fetch() # For the sitemap, as I want all of the urls
  Keywords.query(Keywords.number_articles &gt; 5).fetch() # For the homepage, I want to link to keywords with more than 5 articles
  </code></pre>
  
  <p>Is there a better way to extract data?</p>
  
  <p>I've tried to index data into the Search API, and I've seen huge speed gains. Even though this works, I don't think it's ideal to replicate data from the Datastore into Search API with basically the same fields.</p>
  
  <p>Thanks in advance!</p>