title: MongoDB: cannot iterate through all data with cursor (because data is corrupted)
body: <p>Update:
  The story is <strong>off-topic</strong> and the title <strong>misleading</strong>. The problem is caused by corrupted data set, not cursors, or MongoDB itself. But I would rather like to leave this thread here than to delete it, for that it might help other desperate people.</p>
  
  <p>=== Original story starts here ===</p>
  
  <p>It all starts here: <a href="https://stackoverflow.com/questions/19630795/mongodb-cannot-use-a-cursor-to-iterate-through-all-the-data">MongoDB: cannot use a cursor to iterate through all the data</a></p>
  
  <p>I was trying to iterate through a cursor in Java, and it fails because my collection has too many records(~250M). I tried to allocate a new cursor and use cursor.skip to jump back in when the cursor gets timed out but cursor.skip itself times out.</p>
  
  <p>@mnemosyn pointed out the right way for me: break the job into two stages: In the first stage, use a projected cursor to pull only the monotonic _id's of the records. Record the _id's, and then store it somewhere else as "checkpoints". During the second stage, I can then access any chunk of records as a checkpoint recorded.</p>
  
  <p>So I wrote a javascript like this:</p>
  
  <pre><code>db=connect("localhost/twitter");
  
  db.jobScheduler.drop();
  
  for(var i = 0;i&lt;16;++i)
  {
      db.jobScheduler.save({_id:"s"+i,jobs:[]});
  }
  
  var c = db.tweets.find({},{_id:1}).sort({_id:1});
  
  var totalCount = c.count();
  
  var currentBatchSize = 0;
  var currentNum = 0;
  
  var currentShard = 0;
  var startTid = 0;
  var endTid = 0;
  var currentTid = 0;
  
  while(true)
  {
      while(c.hasNext())
      {
          var doc = c.next()
          currentTid = doc._id;
          if(currentBatchSize == 0)
          {
              startTid = doc._id;
          }
          ++currentNum;
          ++currentBatchSize;
          if(currentBatchSize == 50000)
          {
              currentBatchSize = 0;
              endTid = doc._id;
              db.jobScheduler.update(
                  {_id:"s"+currentShard},
                  {$push:{jobs:[startTid,endTid]}});
              currentShard = (currentShard+1)%16;
              print(currentNum+"/"+totalCount+"("+currentNum*100/totalCount+"%)");
              print("["+startTid+","+endTid+"]");
          }
      }
      if(currentNum != totalCount){
          var c = db.tweets.find({_id:{$gt:currentTid}},{_id:1}).sort({_id:1});
          print("Cursor resetted....");
      }else
          break;
  }
  if(currentBatchSize != 0)
  {
      currentBatchSize = 0;
      endTid = doc._id;
      db.jobScheduler.update(
          {_id:"s"+currentShard},
          {$push:{jobs:[startTid,endTid]}});
      currentShard = (currentShard+1)%16;
  }
  </code></pre>
  
  <p>Considering that simply pulling _id only would still result in timeout, I added a guard like this:</p>
  
  <pre><code>if(currentNum != totalCount){
      var c = db.tweets.find({_id:{$gt:currentTid}},{_id:1}).sort({_id:1});
      print("Cursor resetted....");
  }else
      break;
  </code></pre>
  
  <p>because when the cursor times out, I don't get an exception but a false cursor.hasNext().
  Since I already recorded currentTid when iterating through them, using the range query <code>var c = db.tweets.find({_id:{$gt:currentTid}},{_id:1}).sort({_id:1});</code> will put me back in position, theoretically. However the poor little program end up like this:</p>
  
  <pre><code>[337242463750201340,345999466677010400]
  21800000/253531208(8.598546968624076%)
  [345999469818544100,346244305876295700]
  Cursor resetted....
  Cursor resetted....
  Cursor resetted....
  </code></pre>
  
  <p>It seems to be stuck at the first occurrence of cursor timeout, forever. And the range query is not bringing me back.</p>
  
  <p>Now I'm really confused. Iteration doesn't work. cursor.skip() doesn't work. Range query doesn't work. And what really works with MongoDB? Or is there something I'm doing really wrong?</p>
  
  <p>Any help would be much appreciated!</p>
  
  <p>Update:</p>
  
  <p>I had some discussion with @AsyaKamsky, he helped me to discover the following things:</p>
  
  <ol>
  <li>setting cursor.batchSize() to 10 doesn't work.</li>
  <li>The behaviour is not caused by an idle cursor waiting for 10 minutes. The cursor is pulling data rapidly from server, but still gets invalidated.</li>
  <li>The real problem is that after it gets invalid in this way, I could never reallocate any usable cursors anymore. All the new cursors refuse to give me data. There is one possible workaround: close the cursor before this happens, and re-allocate one and use range query to jump back.</li>
  </ol>
  
  <p>Experiments ongoing. Updating this thread in real-time :-)</p>
  
  <p>Update: Failed! I tried renewing cursor after reading 50k records every time. It also gets trapped at this magical index 21800000! That's very close to my cursor.skip() failure offset!</p>
  
  <p>Update:</p>
  
  <p>Confirmed the guessing:</p>
  
  <pre><code>c = db.tweets.find().skip(21800000); //works
  c = db.tweets.find().skip(21850000); //doesn't work
  </code></pre>
  
  <p>I'll try binary search on this range to find the magic number.</p>
  
  <p>Update:</p>
  
  <p>Ok... Magic number found.</p>
  
  <p>db.tweets.find().itcount()
  ->21837006</p>
  
  <p>db.tweets.find().count()
  ->253531208</p>
  
  <p>Now what? This is really bad.</p>