title: Insert performance of node-mongodb-native
body: <p>I'm testing performance of Node.js with MongoDB.  I know each of these is fine independent of the other, but I'm trying a handful of tests to get a feel for them.  I ran across this issue and I'm having trouble determining the source.</p>
  
  <p><strong>The Problem</strong></p>
  
  <p>I'm trying to insert 1,000,000 records in a single Node.js program.  <strong><em>It absolutely crawls.</em></strong>  We're talking 20 minute execution time.  This occurs whether it's my Mac or CentOS, although the behavior is marginally different between the two.  It does eventually complete.</p>
  
  <p>The effect is similar to swapping, although it's not (memory never exceeds 2 GB).  There are only 3 connections open to MongoDB, and most of the time there's no data being inserted.  It appears to be doing a lot of context switching, and the Node.js CPU core is maxed out.</p>
  
  <p>The effect is similar to the one mentioned in <a href="https://stackoverflow.com/questions/8045838/importing-a-very-large-record-set-into-mongodb-using-nodejs">this thread</a>.</p>
  
  <p>I try the same using PHP and it finishes in 2-3 minutes. No drama.</p>
  
  <p>Why?</p>
  
  <p><strong>Possible Causes</strong></p>
  
  <p>I currently believe this is either a Node.js socket issue, something going on with libev behind the scenes, or some other node-mongodb-native issue.  I may be totally wrong, so I'm looking for a little guidance here.</p>
  
  <p>As for other Node.js MongoDB adapters, I have tried Mongolian and it appears to queue documents in order to batch insert them, and it ends up running out of memory.  So that's out.  (Side note: I have no idea why on this, either, since it doesn't even come close to my 16 GB box limit--but I haven't bothered investigating much further on that.)</p>
  
  <p>I should probably mention that I did in fact test a master/worker cluster with 4 workers (on a quad-core machine) and it finished in 2-3 minutes.</p>
  
  <p><strong>The Code</strong></p>
  
  <p>Here's my Node.js CoffeeScript program:</p>
  
  <pre><code>mongodb = require "mongodb"
  microtime = require "microtime"
  crypto = require "crypto"
  
  times = 1000000
  server = new mongodb.Server "127.0.0.1", 27017
  db = mongodb.Db "test", server
  db.open (error, client) -&gt;
    throw error if error?
  
    collection = mongodb.Collection client, "foo"
  
    for i in [0...times]
      console.log "Inserting #{i}..." if i % 100000 == 0
  
      hash = crypto.createHash "sha1"
      hash.update "" + microtime.now() + (Math.random() * 255 | 0)
      key = hash.digest "hex"
  
      doc =
        key: key,
        foo1: 1000,
        foo2: 1000,
        foo3: 1000,
        bar1: 2000,
        bar2: 2000,
        bar3: 2000,
        baz1: 3000,
        baz2: 3000,
        baz3: 3000
  
      collection.insert doc, safe: true, (error, response) -&gt;
        console.log error.message if error
  </code></pre>
  
  <p>And here's the roughly equivalent PHP program:</p>
  
  <pre><code>&lt;?php
  $mongo = new Mongo();
  $collection = $mongo-&gt;test-&gt;foo;
  
  $times = 1000000;
  for ($i = 0; $i &lt; $times; $i++) {
      if ($i % 100000 == 0) {
          print "Inserting $i...\n";
      }
  
      $doc = array(
          "key" =&gt; sha1(microtime(true) + rand(0, 255)),
          "foo1" =&gt; 1000,
          "foo2" =&gt; 1000,
          "foo3" =&gt; 1000,
          "bar1" =&gt; 2000,
          "bar2" =&gt; 2000,
          "bar3" =&gt; 2000,
          "baz1" =&gt; 3000,
          "baz2" =&gt; 3000,
          "baz3" =&gt; 3000
      );
      try {
          $collection-&gt;insert($doc, array("safe" =&gt; true));
      } catch (MongoCursorException $e) {
          print $e-&gt;getMessage() . "\n";
      }
  }
  </code></pre>