title: retrieve large number of records with mongoDB in a reasonable time
body: <p>I'm using mongoDB to store a querylog and get some stats about it.
  Objects that I store in mongoDB contains the text of the query, the date,
  the user, if the user clicked on some results etc etc. </p>
  
  <p>Now i'm trying to retrieve all the queries not clicked by a user in a certain day 
  with java. My code is approximately this:</p>
  
  <pre><code>    DBObject query = new BasicDBObject();
      BasicDBObject keys = new BasicDBObject();
      keys.put("Query", 1);
      query.put("Date", new BasicDBObject("$gte", beginning.getTime()).append("$lte", end.getTime()));
      query.put("IsClick", false);
      ...
      DBCursor cur = mongoCollection.find(query, keys).batchSize(5000);
  </code></pre>
  
  <p>The output of the query contains about 20k records that I need to iterate. 
  The <strong>problem</strong> is that it takes minutes :( . I don't think is normal.
  From the server log i see:</p>
  
  <pre><code>Wed Nov 16 16:28:40 query db.QueryLogRecordImpl ntoreturn:5000 reslen:252403 nscanned:59260 { Date: { $gte: 1283292000000, $lte: 1283378399999 }, IsClick: false }  nreturned:5000 2055ms
  Wed Nov 16 16:28:40 getmore db.QueryLogRecordImpl cid:4312057226672898459 ntoreturn:5000 query: { Date: { $gte: 1283292000000, $lte: 1283378399999 }, IsClick: false }  bytes:232421 nreturned:5000 170ms
  Wed Nov 16 16:30:27 getmore db.QueryLogRecordImpl cid:4312057226672898459 ntoreturn:5000 query: { Date: { $gte: 1283292000000, $lte: 1283378399999 }, IsClick: false }  bytes:128015 nreturned:2661 --&gt; 106059ms
  </code></pre>
  
  <p>So retrieving the first chunk takes 2 seconds, the second 0.1 seconds, the third 106 seconds!!! weird.. 
  I tried changing the batch size, creating indexes on Date and IsClick, rebooting the machine  :P but no way. Where I'm wrong? </p>