title: nodejs: read from file and store to db, limit maximum concurrent db operations
body: <p>I have a CSV file that I am reading in as a stream, and using transforms to convert to JSON and then asynchronously store each line to a DB.</p>
  
  <p>The issue is that reading from the file is fast, and so leads to very large numbers of concurrent async DB operations, which causes the app to grind to a halt.</p>
  
  <p>I'd like to limit the app such that a max of N outstanding DB operations are in progress at any given time.</p>
  
  <p>This is the basic core of my _transform function:</p>
  
  <pre><code>parser._transform = function(data, encoding, done) {
      //push data rows
      var tick = this._parseRow(data);
  
      //Store tick
      db.set(tick.date, tick, function(err, result) {
        console.log(result);
        if(err) throw err;
      });
  
      this.push(tick);
      done();
  };
  </code></pre>
  
  <p>I've looked at a few options, but these seemed the best candidates:</p>
  
  <ul>
  <li>Use the async api 'forEachLimit'
  <ul>
  <li>The problem I see here is that in my stream transform, I am only operating on one object (line from file) when issuing operations.</li>
  <li>Reading whole file in isn't feasible due to size</li>
  </ul></li>
  <li>Use an asynchronous, parallel, concurrency limited solution as described here, in section 7.2.3:
  <ul>
  <li><a href="http://book.mixu.net/node/ch7.html" rel="nofollow">http://book.mixu.net/node/ch7.html</a></li>
  <li>The problem for me here is what to do in the case 'limit' is reached.</li>
  <li>Spinning or using setTimeout seems to use up all scheduled time and prevents my DB callbacks which should decrement the 'running' counter being initiated.</li>
  </ul></li>
  </ul>
  
  <p>This was my initial attempts at the 'concurrency limited solution':</p>
  
  <pre><code>var limit = 100;
  var running = 0;
  
  parser._transform = function(data, encoding, done) {
    //push data rows
    var tick = this._parseRow(data);
  
    this.push(tick);
    //Store tick to db
    if (running &lt; limit) {
      console.log("limit not reached, scheduling set");
      running++;
      cb.set(tick.date, tick, function(err, result) {
        running--;
        console.log("running is:" + running);
        console.log(result);
        if(err) throw err;
      });
    } else {
      console.log("max limit reached, sleeping");
      setTimeout(this._transform(data, encoding, done),1000);
    }
    done();
  };
  </code></pre>
  
  <p>I've only started node.js this week, so I'm not clear what the correct model for solving this is.</p>
  
  <p><strong>Note:</strong> Couple of things I am aware of that is that this should at least be an exponential backoff if using the latter model, and there should be some 'max backoffs' system in place, so as not to blow out the call stack. Tried to keep it simple here for now though.</p>