title: simple PHP web crawler working on some, certain types of, pages
body: <p>I made this simple PHP web crawler that gets the source from a page after the opening body tag, strips the other HTML tags, and then echo's the content.</p>
  
  <p>It works when I initiate it giving it a page ending in .html, but when I give in a URL like the URL to a set of results from Google, it doesn't follow those links and get the content and echo the content.</p>
  
  <p>How can I get it to follow the URL of a Google search result and follow the links within and echo their content?</p>
  
  <p>Here is the code of the crawler:</p>
  
  <pre><code>error_reporting( E_ERROR );
  
  define( "CRAWL_LIMIT_PER_DOMAIN", 50 );
  
  $domains = array();
  
  $urls = array();
  
  $dom = new DOMDocument();
  
  $matches = array();
  
  function crawl( $domObject, $url, $matchList )
  {
      global $domains, $urls;
      $parse = parse_url( $url );
      $domains[ $parse['host'] ]++;
      $urls[] = $url;
  
      $content = file_get_contents( $url );
      if ( $content === FALSE ){
           return;
  }
  
      $content = stristr($content, "&lt;body&gt;");
      $domObject-&gt;loadHTML($content);
      $anchors = $domObject-&gt;getElementsByTagName('a');
      foreach($anchors as $anchor){
           if(preg_match('/(?:https?:\/\/|www)[^\'\" ]*/i', (string)($anchor-&gt;getAttribute('href')))){
               array_push($matchList, (string)($anchor-&gt;getAttribute('href')));
           }
           else{
               preg_match('/(?:https?:\/\/|www)[^\/]+(?:\S*?\/)*/i', $url, $beginings);
               $urlPrefix = $beginings[0];
               $absolute = (string)(((string)$urlPrefix).((string)$anchor-&gt;getAttribute('href')));
               array_push($matchList, $absolute);
           }
       }
       echo  strip_tags($content) . "&lt;br /&gt;&lt;br /&gt;&lt;br /&gt;";
  
       foreach( $matchList as $crawled_url ) {
           $parse = parse_url( $crawled_url );
           if ( count( $domains[ $parse['host'] ] ) &lt; CRAWL_LIMIT_PER_DOMAIN &amp;&amp; !in_array( $crawled_url, $urls ) ) {
              sleep( 1 );
              crawl( $domObject, $crawled_url, $matchList );
           }
        }
   }
  
   crawl($dom, 'http://www.google.com/search?q=google', $matches);
  </code></pre>