title: How to return large amount of rows from mongodb using node.js http server?
body: <p>I have a user database in mongodb which I would like to export via a REST interface in JSON. The problem is that in the worst case scenario the amount of returned rows is well over 2 million.</p>
  
  <p>First I tried this</p>
  
  <pre class="lang-js prettyprint-override"><code>var mongo = require('mongodb'),
    Server = mongo.Server,
    Db = mongo.Db;
  var server = new Server('localhost', 27017, {auto_reconnect: true});
  var db = new Db('tracking', server);
  var http = require('http');
  
  http.createServer(function (request, response) {
    db.collection('users', function(err, collection) {
      collection.find({}, function(err, cursor){
        cursor.toArray(function(err, items) {
          output = '{"users" : ' + JSON.stringify(items) + '}';
  
          response.setHeader("Content-Type", "application/json");
          response.end(output);
        });
      });
    });
  }).listen(8008);
  console.log('Server running at localhost:8008');
  </code></pre>
  
  <p>which fails when running out of memory. The example uses node-mongodb-native driver and the basic http package.</p>
  
  <blockquote>
    <p>FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory</p>
  </blockquote>
  
  <p>(note that in real scenario I use parameters which limit the results as needed, but this example queries them all which is the worst case scenario regardless)</p>
  
  <p>The data itself is simple, like</p>
  
  <blockquote>
    <p>{ "_id" : ObjectId("4f993d1c5656d3320851aadb"), "userid" : "80ec39f7-37e2-4b13-b442-6bea57472537", "user-agent" : "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322)", "ip" : "127.0.0.1", "lastupdate" : 1335442716 }</p>
  </blockquote>
  
  <p>I also tried something like </p>
  
  <pre class="lang-js prettyprint-override"><code>while(cursor != null)
  {
    cursor.nextObject(function(err, item) {
      response.write(JSON.stringify(item));
    });
  }
  </code></pre>
  
  <p>but that ran out of memory too.</p>
  
  <p>How should I proceed? There should be a way to stream the data row by row, but I haven't been able to find a suitable example for it. Paging the data is out of the question because of external application requirements. I thought of writing the data to a file and then posting it, but that leads to unwanted io.</p>