title: Node - Large for loops with database work running out of memory
body: <p>So I have a large amount of data I need to place into database after I generated it all. Now whether I attempt to do this while the data is being generated or after, doesnt matter. It crashes both ways</p>
  
  <p>I have code like this:</p>
  
  <pre><code>for(var i = 0; i&lt;4000, i++){
      for(var z = 0; z&lt;1000; z++){
          c.query("REPLACE INTO `"+i+"`VALUES ("+z+", "+data+",)", function (err){
              if(err){
                  throw err;
              }
          });
      }
  }
  </code></pre>
  
  <p>After about 100 loops of the outer for loop, it gets killed. When i look at the memory info, its running out of memory. </p>
  
  <p>I can get better results (it makes it to about 1500 of the outerloop) if i change it all to self-calling functions with a delay on the call for about 100ms. I assume that's because the event queue has time to clear out. </p>
  
  <p>I've tried this in every way I know how so far. I've tried redis and now mysql, neither work. It does work with a json file database but not these. </p>
  
  <p>Anyone have any ideas of how I can make this loop through. This is a one time task, so speed is of no concern at all, i just need it to complete lol.  </p>
  
  <p>Edit: This is in right direction, but keep gettin Max call stack size exceeded. Can't figure it out.</p>
  
  <pre><code>async.eachOfSeries(chunk, function(item, key, outerCallback){
      async.eachOfSeries(item, function(value, slot, innerCallback){
          //console.log(slot);
          c.query("REPLACE INTO `"+key+"` (r, v) VALUES ("+slot+", "+value+")", function(err){
              if(err){
                  throw err;
              }
          });
          innerCallback();
      }, function(err){
          outerCallback();
      });
  }, function(err){
      callback();
  });
  </code></pre>
  
  <p>I tried setting a delay to the callbacks, but no luck. </p>