title: How do I best handle this pagination with asynchronous Node.js?
body: <p>My question involves the use of asynchronous code.</p>
  
  <p>Here's the relevant pseudocode:</p>
  
  <pre><code>function handleQueryMore(result) {
      conn.queryMore(result.nextRecordsUrl, function(err, resultMore){    
          if (err) {}
          //do stuff with result
          if (!resultMore.done){
              handleQueryMore(resultMore);
          }
      });
  }
  
  //(below code runs first when launched of course)
  
  var conn = new jsforce.Connection({////etc
  var sql = "SELECT //.......
  
  conn.query(sql, function(err, result) {
      if (err) {}
      //do stuff with result
  
      if (!result.done) //didn't pull all records
      {
          handleQueryMore(result);
      }               
  });
  </code></pre>
  
  <p>The initial query only returns a certain maximum # of records and then handleQueryMore() is called recursively to handle each additional chunk of records.</p>
  
  <p>Originally I had a single <code>conn.query()</code> in a where <code>(!result.done){}</code> loop but of course the problem there was that the <code>conn.query()</code> code runs asynchronously and doesn't get a chance to finish before the next run of the loop (resulting in an ~infinite purposeless loop).</p>
  
  <p>There are 3rd party library ways to get the code to run syncronously but my guess is that there is some fundamental design paradigm that I'm not following.  The recursion works but I fear the amount of memory that could be needed if the query returns a huge # of records.</p>
  
  <p>Even if I know the # of records I need to paginate through, I MUST have the <code>result.nextRecordsUrl</code> to paginate, which I can't get until each previous query is executed... so I can't just run them all at the same time.</p>
  
  <p>Anyone care to weigh in on this?</p>
  
  <p>Thanks!</p>