title: Insert streamed XML data database
body: <p>I am trying to efficiently insert a lot of data (XML file is over 70GB in size) without crashing my MongoDB server. Currently this is what I am doing using <a href="https://github.com/assistunion/xml-stream" rel="nofollow noreferrer"><code>xml-stream</code></a> in NodeJS:</p>
  
  <pre><code>var fs = require('fs'),
      path = require('path'),
      XmlStream = require('xml-stream'),
      MongoClient = require('mongodb').MongoClient,
      assert = require('assert'),
      ObjectId = require('mongodb').ObjectID,
      url = 'mongodb://username:password@my.server:27017/mydatabase',
      amount = 0;
  
  var stream = fs.createReadStream(path.join(__dirname, 'motor.xml'));
  var xml = new XmlStream(stream);
  
  xml.collect('ns:Statistik');
  xml.on('endElement: ns:Statistik', function(item) {
      var insertDocument = function(db, callback) {
          db.collection('vehicles').insertOne(item, function(err, result) {
              amount++;
              if (amount % 1000 == 0) {
                  console.log("Inserted", amount);
              }
              callback();
          });
      };
  
      MongoClient.connect(url, function(err, db) {
          insertDocument(db, function() {
              db.close();
          });
      });
  });
  </code></pre>
  
  <p>When I call the <code>xml.on()</code> it basically returns the tree/element I am currently on. Since this is straight up JSON, I can just give it to my <code>db.collection().insertOne()</code> function as a parameter and it will insert it into the database exactly how I want it.</p>
  
  <p>All of the code actually works as it is now, but it stops after about 3000 insertions (which takes about 10 seconds). I suspect it's because I open a database connection, insert the data, and then close the connection each and every time I see a tree in the XML file, in this case about 3000 times.</p>
  
  <p>I could, somehow, incorporate the <code>insertMany()</code> function and do it in chunks of 100s (or more), but I'm not quite sure how that would work with it all being streamed and asynchronous.</p>
  
  <p>So my question is: How do I insert a large amount of XML (to JSON) into my MongoDB database without it crashing?</p>