title: PHP Scraping company data using Guzzle
body: <p>I'm trying to audit a vast amount of company data from companycheck.co.uk my current script appears to be looping the first 10 results from only the first page. I had the script gather more than 10 results at one point, but this caused a fatal error after around 600 results (not a timeout error, but a connection error of some sort), I need the script to be more reliable as I'm fetching over 40,000 results.</p>
  
  <p>My code so far:</p>
  
  <pre><code>&lt;?php
  set_time_limit(0);
  ini_set('max_execution_time', 0);
  
  require 'vendor/autoload.php';
  require "Guzzle/guzzle.phar";
  
  
  // Add this to allow your app to use Guzzle and the Cookie Plugin.
  use Guzzle\Http\Client as GuzzleClient;
  use Guzzle\Plugin\Cookie\Cookie;
  use Guzzle\Plugin\Cookie\CookiePlugin;
  use Guzzle\Plugin\Cookie\CookieJar\ArrayCookieJar;
  use Guzzle\Plugin\Cookie\CookieJar\CookieJarInterface;
  
  
  
  $Pagesurl  = 'http://companycheck.co.uk/search/UpdateSearchCompany?searchTerm=cars&amp;type=name';
  $pagesData = json_decode(file_get_contents($Pagesurl), true);
  
  
  $resultsFound = $pagesData["hits"]["found"];
  $pages        = ceil($resultsFound / 10);
  //echo $pages;
  
  echo "&lt;br&gt;";
  
  for ($p = 0; $p &lt; $pages; $p++) {
  
      $url  = 'http://companycheck.co.uk/search/UpdateSearchCompany?searchTerm=cars&amp;type=name&amp;companyPage=' . $p . '';
      $data = json_decode(file_get_contents($url), true);
      for ($i = 0; $i &lt; 11; $i++) {
          $id             = $data["hits"]["hit"][$i]["id"];
          $TradingAddress = $data["hits"]["hit"][$i]["data"]["address"][0];
          $companyName    = $data["hits"]["hit"][$i]["data"]["companyname"][0];
          $companyNumber  = $data["hits"]["hit"][$i]["data"]["companynumber"][0];
  
          $finalURL = "http://companycheck.co.uk/company/" . $id . "";
  
  
          $httpClient = new GuzzleClient($finalURL);
          $httpClient-&gt;setSslVerification(FALSE);
  
          $cookieJar    = new ArrayCookieJar();
          // Create a new cookie plugin
          $cookiePlugin = new CookiePlugin($cookieJar);
          // Add the cookie plugin to the client
          $httpClient-&gt;addSubscriber($cookiePlugin);
          $httpClient-&gt;setUserAgent("Opera/9.23 (Windows NT 5.1; U; en-US)");
  
          $request  = $httpClient-&gt;get($finalURL);
          $response = $request-&gt;send();
  
          $body    = $response-&gt;getBody(true);
          $matches = array();
  
          preg_match_all('/&lt;table.*?&gt;(.*?)&lt;\/table&gt;/si', $body, $table);
          preg_match('/&lt;meta name=\"keywords\" content=\"(.*?)\"\/&gt;/si', $body, $metaName);
          preg_match('/&lt;p itemprop="streetAddress".*?&gt;(.*?)&lt;\/p&gt;/si', $body, $regOffice);
  
  
          echo "&lt;table&gt;&lt;tbody&gt;";
  
          echo "&lt;tr&gt;&lt;th&gt;Company Name&lt;/th&gt;&lt;td&gt;";
          echo $companyName;
          echo "&lt;/td&gt;&lt;/tr&gt;";
  
          echo "&lt;tr&gt;&lt;th&gt;Company Number&lt;/th&gt;&lt;td&gt;";
          echo $companyNumber;
          echo "&lt;/td&gt;&lt;/tr&gt;";
  
          echo "&lt;tr&gt;&lt;th&gt;Registar Address&lt;/th&gt;&lt;td&gt;";
          echo str_replace("&lt;br&gt;", " ", $regOffice[0]);
          echo "&lt;/td&gt;&lt;/tr&gt;
  
  &lt;tr&gt;&lt;th&gt;Trading Address&lt;/th&gt;&lt;td&gt;";
          echo $TradingAddress;
          echo "&lt;/td&gt;&lt;/tr&gt;
  &lt;tr&gt;
  &lt;th&gt;Director Name&lt;/th&gt;
  &lt;td&gt;";
          $name = explode(',', $metaName[1]);
          echo $name[2];
          echo "&lt;/td&gt;
  
  &lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;";
          echo $table[0][1];
          echo "&lt;br&gt;&lt;br&gt;&lt;br&gt;";
  
  
      }
  
  }
  </code></pre>
  
  <p>To get each page, I use <a href="http://companycheck.co.uk/search/UpdateSearchCompany?searchTerm=cars&amp;type=name&amp;companyPage=1" rel="nofollow">http://companycheck.co.uk/search/UpdateSearchCompany?searchTerm=cars&amp;type=name&amp;companyPage=1</a> which returns json for each page from <a href="http://companycheck.co.uk/search/results?SearchCompaniesForm[name]=cars&amp;yt1=" rel="nofollow">http://companycheck.co.uk/search/results?SearchCompaniesForm[name]=cars&amp;yt1=</a> and some data, but not all.</p>
  
  <p>With this I can get the ID of each company to navigate to each link and scrape some data from the frontend of the site.</p>
  
  <p>For example the first result is:</p>
  
  <pre><code>"hits":{"found":42842,"start":0,"hit":[{"id":"08958547","data":{"address":["THE ALEXANDER SUITE SILK POINT, QUEENS AVENUE, MACCLESFIELD, SK10 2BB"],"assets":[],"assetsnegative":[],"cashatbank":[],"cashatbanknegative":[],"companyname":["CAR2CARS LIMITED"],"companynumber":["08958547"],"dissolved":["0"],"liabilities":[],"liabilitiesnegative":[],"networth":[],"networthnegative":[],"postcode":["SK10 2BB"],"siccode":[]}}
  </code></pre>
  
  <p>So the first link is: <a href="http://companycheck.co.uk/company/08958547" rel="nofollow">http://companycheck.co.uk/company/08958547</a></p>
  
  <p>Then from this I can pull table data such as:</p>
  
  <pre><code>Registered Office
  
  THE ALEXANDER SUITE SILK POINT
  QUEENS AVENUE
  MACCLESFIELD
  SK10 2BB
  </code></pre>
  
  <p>And information from the meta tags such as:</p>
  
  <pre><code>&lt;meta name="keywords" content="CAR2CARS LIMITED, 08958547,INCWISE COMPANY SECRETARIES LIMITED,MR ROBERT CARTER"/&gt;
  </code></pre>
  
  <p>An example of one of the results returned:</p>
  
  <pre><code>Company Name    CAR2CARS LIMITED
  Company Number  08958547
  Registar Address    
  THE ALEXANDER SUITE SILK POINT QUEENS AVENUE MACCLESFIELD SK10 2BB
  
  Trading Address THE ALEXANDER SUITE SILK POINT, QUEENS AVENUE, MACCLESFIELD, SK10 2BB
  Director Name   INCWISE COMPANY SECRETARIES LIMITED
  Telephone   No telephone number available.
  Email Address   No email address available.
  Contact Person  No contact person available.
  Business Activity   No Business Activity on record.
  </code></pre>
  
  <p>Each json page contains 10 company IDs to put into the URL to find the company, from each of these companies I need to scrape data from the full URL, then after these 10 move onto the next page and get the next 10 and loop this up until the last page.</p>