title: Scraping attempts getting 403 error
body: <p>I am trying to scrape a website and I am getting a 403 Forbidden error no matter what I try:</p>
  
  <ol>
  <li>wget</li>
  <li>CURL (command line and PHP)</li>
  <li>Perl WWW::Mechanize</li>
  <li>PhantomJS</li>
  </ol>
  
  <p>I tried all of the above with and without proxies, changing user-agent, and adding a referrer header. </p>
  
  <p>I even copied the request header from my Chrome browser and tried sending with my request using PHP Curl and I am still getting a 403 Forbidden error.</p>
  
  <p>Any input or suggestions on what is triggering the website to block the request and how to bypass?</p>
  
  <p>PHP CURL Example:</p>
  
  <pre><code>$url ='https://www.vitacost.com/productResults.aspx?allCategories=true&amp;N=1318723&amp;isrc=vitacostbrands%3aquadblock%3asupplements&amp;scrolling=true&amp;No=40&amp;_=1510475982858';
  $headers = array(
              'accept:application/json, text/javascript, */*; q=0.01',
              'accept-encoding:gzip, deflate, br',
              'accept-language:en-US,en;q=0.9',               
              'referer:https://www.vitacost.com/productResults.aspx?allCategories=true&amp;N=1318723&amp;isrc=vitacostbrands:quadblock:supplements',
              'user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36',
              'x-requested-with:XMLHttpRequest',
  );
  
  $res = curl_get($url,$headers);
  print $res;
  exit;
  
  function curl_get($url,$headers=array(),$useragent=''){ 
      $curl = curl_init();
      curl_setopt($curl, CURLOPT_URL, $url);
      curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
      curl_setopt($curl, CURLOPT_HEADER, true);           
      curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
      curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
      curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);   
      curl_setopt($curl, CURLOPT_ENCODING, '');            
      if($useragent)curl_setopt($curl, CURLOPT_USERAGENT,$useragent);             
      if($headers)curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
  
      $response = curl_exec($curl);       
  
      $header_size = curl_getinfo($curl, CURLINFO_HEADER_SIZE);
      $header = substr($response, 0, $header_size);
      $response = substr($response, $header_size);
  
  
      curl_close($curl);  
      return $response;
   }
  </code></pre>
  
  <p>And here is the response I always get:</p>
  
  <pre><code>&lt;HTML&gt;&lt;HEAD&gt;
  &lt;TITLE&gt;Access Denied&lt;/TITLE&gt;
  &lt;/HEAD&gt;&lt;BODY&gt;
  &lt;H1&gt;Access Denied&lt;/H1&gt;
  
  You don't have permission to access     
  
    "http&amp;#58;&amp;#47;&amp;#47;www&amp;#46;vitacost&amp;#46;com&amp;#47;productResults&amp;#46;aspx&amp;#63;" 
  on this server.&lt;P&gt;
  Reference&amp;#32;&amp;#35;18&amp;#46;55f50717&amp;#46;1510477424&amp;#46;2a24bbad
  &lt;/BODY&gt;
  &lt;/HTML&gt;
  </code></pre>