title: PHP Curl request returns unsuccessful, how to circumvent robots.txt?
body: <p>I have a scraper that used to work, but now it is returning a 'request unsuccessful' page, presumably because of changed robots.txt settings. To address ethical issues now - I plan to switch to a different site, as this one updated their TOS and no longer likes crawlers. It is only a simple scraper and I can build a new one in an hour. However, I am interested in this from a technical standpoint, as I thought Curl was supposed to be able to mimic a browser request, and the site opens fine through a browser.</p>
  
  <p>However, this is what I get instead of the page content when using Curl:</p>
  
  <pre><code>&lt;html style="height:100%"&gt;&lt;head&gt;&lt;META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"&gt;&lt;meta name="format-detection" content="telephone=no"&gt;&lt;meta name="viewport" content="initial-scale=1.0"&gt;&lt;meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"&gt;&lt;/head&gt;&lt;body style="margin:0px;height:100%"&gt;&lt;iframe src="/_Incapsula_Resource?CWUDNSAI=9&amp;xinfo=7-43571122-0 0NNY RT(1459673434797 0) q(0 -1 -1 -1) r(0 -1) B12(4,315,0) U2&amp;incident_id=133001530113712368-463745872568846743&amp;edet=12&amp;cinfo=04000000" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px"&gt;Request unsuccessful. Incapsula incident ID: 133001530113712368-463745872568846743&lt;/iframe&gt;&lt;/body&gt;&lt;/html&gt;
  </code></pre>
  
  <p>The same is received when I try a simple <code>file_get_contents</code> instead of Curl.</p>
  
  <p>This is my Curl code:</p>
  
  <pre><code>        $curl = curl_init();
          curl_setopt_array($curl, array(
              CURLOPT_RETURNTRANSFER =&gt; 1,
              CURLOPT_URL =&gt; $url,
              CURLOPT_HTTPHEADER     =&gt; [
              'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0',
              'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
              ]
          ));
          $raw = curl_exec($curl);
          curl_close($curl);
  </code></pre>
  
  <p>There is suprisingly little on this issue online. Based on some similar ones I found, I have also tried adding <code>CURLOPT_USERAGENT =&gt; 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5'</code>, <code>CURLOPT_SSL_VERIFYPEER =&gt; 0</code>, <code>CURLOPT_SSLVERSION =&gt; 3</code> but to no avail.</p>
  
  <p>I have also tried both <code>http</code> and <code>https</code> in the target URL - in the browser is seems to load <code>https</code> automatically, though I am not sure if this was always the case.</p>
  
  <p>Does anyone know why Curl is failing and what might be the solution?</p>