title: How to implement the logic of this asynchronous nodejs web scraping program?
body: <p>I currently have a nodejs based web scraper that utilities the puppetteer module. While it does work, it is very slow, since I have made it in such a way that it uses a synchronous approach instead of an asynchronous one.</p>
  <p>The basic logic of the program in pseudo code is as follows:</p>
  <pre><code>async fucntion main():
      
     ......
  
      while true:
          for url in listOfUrls:
              await scrapeInformation()
              if there is a change:
                  sendNotification()
  </code></pre>
  <p>The problem with this approach is that I can not begin the scraping of another page until the current page has been scraped. I would like to begin the loading of the next webpages, so that they are ready to be scraped once their turn comes in the  <code>for</code> loop. However, I still want to be able to limit the number of webpages open for scraping, so that I do not run into any memory errors, since I ran into that issue in a previous implementation of this script where I was launching instances of the chromium browser much faster than the program was able to close them.</p>
  <p>The scrapeInformation() looks a bit like this:</p>
  <pre><code>async function scrapeInformation(url, browser) {
      const browser = await puppeteer.launch({headless: true});
      const page = await browser.newPage();
      let response = await page.goto(url);
  
      let data = await page.evaluate(() =&gt; {
  
          blah blah blah
          
          return {blah, blah};
      });
  
      await page.close();
      
      return data
  }
  </code></pre>
  <p>I believe a good place to start would be to perhaps to rescrape another URL at the <code>let data = await page.evaluate(() =&gt; {</code> line, but I am unsure as how to implement such logic.</p>