title: How to avoid - Error 403 while web scraping using cheerio
body: <p>I'm web scraping a website and I have an array of links:</p>
  <pre><code> 'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Abercorn',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Longueuil',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Sainte-Anne-De-Bellevue',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Shawinigan',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Chateauguay',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Mont-Laurier',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Georges',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Sherbrooke',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Chicoutimi',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Montreal',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Henri-De-Levis',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Stukely-Sud',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Drummondville',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Montreal-Est',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Hubert',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Trois-Rivieres',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Gatineau',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Montreal-Nord',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Jerome',
    &quot;http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Val-D'or&quot;,
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Granby',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Montreal-Ouest',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Lambert',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Verdun',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Lachine',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Quebec',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Laurent',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Warwick',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Lasalle',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Rigaud',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Leonard',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Westmount',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Laval',
    'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Roxboro'
  </code></pre>
  <p>But when I do the request some of those links return error 403 - Forbidden.</p>
  <pre><code>Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Verdun
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Granby
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Lambert
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Val-D'or
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Lasalle
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Laval
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Warwick
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Quebec
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Westmount
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Roxboro
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Laurent
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Rigaud
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Saint-Leonard
  Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&amp;StateProv=QC&amp;City=Lachine
  </code></pre>
  <p>When I use a list with fewer links it works perfectly.</p>
  <p>Here is my code:</p>
  <pre><code> const request = require('request');
    const cheerio = require('cheerio');
        
    function readChurches(cities){
          const churches = []
          for (let index = 0; index &lt; cities[0].length; index++){
              const city = cities[0][index];
              churches.push(new Promise((resolve, reject) =&gt; {
                  const church = [] 
                  let options = {
                      url: city,
                      headers: {
                           'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
                      }
                  };
                  
                  request(options, (error, response, html) =&gt; {
                      if(!error &amp;&amp; response.statusCode == 200) {
                          const $ = cheerio.load(html);
                          const $$ = cheerio.load($('table').find('tbody').eq(1).find('tr').eq(1).find('td').eq(1).html())
          
                          $$('a').each((i, el) =&gt; {
                              const item = $(el).attr('href')
                              if(item != undefined){
                                  if(item.includes('ViewEntity')) {
                                      church.push(`http://www.adventistdirectory.org${item}`);
                                  }    
                              }
                          });
                          resolve(church);
                      } else {
                          console.log('Error',error,response.statusMessage,city)
                          reject(error)
                      }
                  });
              }))
          }
      
          return Promise.all(churches);
      }
  </code></pre>
  <p>What can be done to bypass error 403?. Because when I try to open the link on my browser it works, when I use the javascript function doesn't work though.</p>
  <p><em><strong>--- NEW UPDATES ---</strong></em></p>
  <p>I've changed to code. I added a try catch block</p>
  <pre><code>function readChurches(cities){
      const churches = []
      for (let index = 0; index &lt; cities[0].length; index++){
          const city = cities[0][index];
          churches.push(new Promise((resolve, reject) =&gt; {
              const church = [] 
              let options = {
                  url: city,
                  headers: {
                       'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
                  }
              };
  
              try {
                  request(options, (error, response, html) =&gt; {
                      if(!error &amp;&amp; response.statusCode == 200) {
                          const $ = cheerio.load(html);
                          const $$ = cheerio.load($('table').find('tbody').eq(1).find('tr').eq(1).find('td').eq(1).html())
          
                          $$('a').each((i, el) =&gt; {
                              const item = $(el).attr('href')
                              if(item != undefined){
                                  if(item.includes('ViewEntity')) {
                                      church.push(`http://www.adventistdirectory.org${item}`);
                                  }    
                              }
                          });
                          resolve(church);
                      } 
                  });
                  
              } catch (error) {
                  console.log('Error',error,city)
                  reject(error)
              }
          }))
      }
  
      return churches
  }
  </code></pre>
  <p>and also, created this function, provided by @chrispytoes</p>
  <pre><code>async function doStuff(churches) {
      const results = [];
      for(let i in churches) {
          try {
              console.log(churches[i])
              results.push(await churches[i]);
              sleep(5000);    
          } catch (error) {
              console.log(error)
          }
      }
  
      return results
  }
  </code></pre>
  <p>and I'm running it:</p>
  <pre><code>async function run(){
      
          let provinces = []
          provinces.push(`http://www.adventistdirectory.org/BrowseStateProv.aspx?CtryCode=CA&amp;StateProv=QC`)
          
          let cities = await readCities(provinces);
      
          const churches = await readChurches(cities);
          const stuff = await doStuff(churches)
          
          console.log('Churches: ', stuff);
      
          console.log('End')
          
      } catch (error) {
          console.log('Error', error)
      }
  }
  </code></pre>
  <p>And I'm getting this on my console:</p>
  <pre><code>Promise { &lt;pending&gt; }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19653'
    ]
  }
  Promise { &lt;pending&gt; }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19637'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=54633'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=31155'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15271'
    ]
  }
  Promise { &lt;pending&gt; }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=30783'
    ]
  }
  Promise { &lt;pending&gt; }
  Promise { &lt;pending&gt; }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15265'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15255'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15251'
    ]
  }
  Promise { &lt;pending&gt; }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15247'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19645'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=32838'
    ]
  }
  Promise {
    [
      'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=29973'
    ]
  }
  Promise { &lt;pending&gt; }
  </code></pre>
  <p>it is not getting to <code>console.log('Churches: ', stuff);</code></p>