Title: Nodejs - Making a very large 'X' amount of calls to a function that returns a Promise one at a time
Tags: node.js;asynchronous;promise;es6-promise;synchronous
Question: I have a use case where I need to call an asynchronous NPM library many (>100,000 times) in a serial manner. This is for a compliance/research-related task in a program that will be running overnight (runtime is not super important). I am not able to make concurrent calls to the library from the same machine. That being said, I do not need to waterfall previous responses into later ones...the requests are completely independent of each other with the only constraint being that the library cannot handle concurrent requests from the same machine.
At a high level, I have my NPM library call as a function returning a Promise. My issue is that I do not know how to write a program that will handle sending that many requests one at a time in an automated fashion. If I were dealing with a known number of small (<5) requests, I could simply explicitly chain them together with .then() statements or even use a callback nest instead of Promises. If it were a larger, unknown number much less than the N > 100,000 I am dealing with now, I would write a function that could recursively call itself in the .then(). The recursive approach is what I'm doing now, which is suitable for a sample size of around 500 but will never work going 100,000 or more deep. My question is: How can I write a program that will automatically handle making these requests in a synchronous fashion without using recursion?
Here is an abstracted sample of the recursive solution I am currently using for my much smaller sample size of requests. inputs is a list containing information for each individual NPM call. Each item in inputs corresponds with an call to the NPM library that I must make.
let recursiveRunner = (index) =&gt; {
// basecase
if (index &gt;= inputs.length) {
// do stuff at end
return;
let input = inputs[index];
myPromisifiedNpmCall(input)
.then(result =&gt; {
// store my result appropriately
someStoreResultFunction(result)
// recursive call
recursiveRunner(index + 1);
})
.catch(err =&gt; {
// do stuff here
});
// kick off recursiveRunner
recursiveRunner(0)
Comment: Why doesn't your current approach scale to the larger problem?

Comment: Uhhh, async recursive has no stack build-up and thus has no limits for how many requests it can serve.  The stack completely unwinds before `.then()` handlers are called (that's in the promise specification) so while it appears to be classic recursion with stack build-up, it isn't really.  If your solution works for 5, it will work for 1,000,000 just fine..

Here is the accepted answer: It seems there's no need for recursion, this can be solved with simple loop. This is a perfect use case for async..await that allows synchronous-like control flow:
async function runner() {
for (let i = 0; i < inputs.length; i++) {
try {
const input = inputs[i];
const result = await myPromisifiedNpmCall(input);
someStoreResultFunction(result);
} catch (err) {...}
Which is roughly equivalent to less resource efficient ES6 code:
function runner() {
let promise = Promise.resolve();
for (let i = 0; i < inputs.length; i++) {
promise = promise
.then(() =&gt; {
const input = inputs[i];
return myPromisifiedNpmCall(input);
})
.then(result =&gt; {
someStoreResultFunction(result);
})
.catch(err =&gt; {...});
return promise;
Comment for this answer: @jfriend00 Thanks for the notice. Yes, that's the difference between them. I added ES6 version for the reference only. I suppose that that's the case that could benefit from recursion.
Comment for this answer: @purdoo For older Node versions that don't support async/await an appropriate solution would be `co` with generators that looks very similar to async/await, roughly what async/await does internally. The problem with second snippet is that a chain of 100000 thens is formed synchronously and occupies RAM which won't be freed until this promise is fulfilled or rejected. This works well for smaller chains that resolve quickly but is inefficient for your case.
Comment for this answer: Switching to Node 8.10 was fine for my use case so the async-await suggestion ended up working perfectly. Also, I had no idea about the lack of stack-buildup nuance here with the Promises. That's good to know...I had assumed it was a typical recursion problem and concluded it wouldn't work with my input size.
Comment for this answer: Your desugared version is a bit different.  If `inputs.length` was 1,000,000, that would build a gigantic promise chain 1,000,000 elements long rather than just running each one one at a time.  That's potentially a lot more peak memory consumption than either the OP's recursive solution of your async/await solution.
Comment for this answer: I guess my point (for the benefit of the OP) is that the OP's (seemingly recursive) solution will work just fine for large numbers without any stack buildup (I commented to them as such) or larger peak memory usage.  Similarly, your `async/await` solution is good in that regard and arguably simpler code to write, understand and maintain.  I'd argue that your ES6 version is not really better than what the OP already has.

Here is another answer: I recommend using a 3rd party library like async series, to have more flow control over your asynchronous tasks. If you need them to run on parallel, then take a look at async parallel.
Comment for this answer: async library is obsolete and cannot be recommended as a rule of thumb. Its core functionality is covered by native promises.
Comment for this answer: Can you provide a link demonstrating your statement and sample code showing that your solution doesn't end in a stack overflow by running the number of iterations (100,000) that OP asked?

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 2, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2018:00:07/17/18', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<node.js><asynchronous><promise><es6-promise><synchronous>', 'Title': "Nodejs - Making a very large 'X' amount of calls to a function that returns a Promise one at a time", 'ViewCount': 137, 'id': 51390041, 'user_AboutMe': '', 'user_AccountId': 4735606.0, 'user_DisplayName': 'purdoo', 'user_Reputation': 932.0, 'user_Views': 58.0, 'user_id': 3830082.0}}