title: Scrape page and separate internal from external links
body: <p>Building a little PHP <strong>scraper</strong> , I'm writing a little function which should separate my Internal &amp; External links,</p>
  <p>I'm passing the function a copy of the html source code along with the base host address</p>
  <pre><code>$source = file_get_contents('http://www.example.com');
  $host   = &quot;mysite.com&quot;;
  </code></pre>
  <p>here is my function so far...</p>
  <pre><code>function find_page_links($source, $host){
  
      if($source){
          
      $htmlDoc = new DomDocument();
      @$htmlDoc-&gt;loadhtml($source);
          
      $int_links = array();
      $ext_links = array();
      
      // GET LINKS
      foreach($htmlDoc-&gt;getElementsByTagName('a') as $link) {
              
          $url   = trim($link-&gt;getAttribute('href'));
          $title = trim($link-&gt;getAttribute('title'));
          $text  = trim($link-&gt;nodeValue);
          $rel   = trim($link-&gt;getAttribute('rel'));
              
          $pos = strpos($url,$host);
              
          if( $pos === false ){ // NO MATCH EXTERNAL              
              if( (substr($url, 0, 1) == '/') || 
                          (substr($url, 0, 1) == '#') )
                      {
              // INTERNAL
              $int_links[] = array( 'link_url' =&gt; $url, 
                                            'link_text' =&gt; $text, 
                                            'link_title' =&gt; $title, 
                                            'link_rel' =&gt; $rel 
                                          );
              }else{
              // EXTERNAL
              $ext_links[] = array( 'link_url' =&gt; $url, 
                                            'link_text' =&gt; $text, 
                                            'link_title' =&gt; $title, 
                                            'link_rel' =&gt; $rel 
                                          );
              }
          }else{
               if( $pos &lt; 20 ){
               // INTERNAL
               $int_links[] = array( 'link_url' =&gt; $url, 
                                         'link_text' =&gt; $text, 
                                         'link_title' =&gt; $title, 
                                         'link_rel' =&gt; $rel );
               }else{
               // EXTERNAL
               $ext_links[] = array( 'link_url' =&gt; $url, 
                                           'link_text' =&gt; $text, 
                                           'link_title' =&gt; $title, 
                                           'link_rel' =&gt; $rel 
                                          );  
               }
          } // end else
      } // end foreach
          
      $content = array();
      $content['int_links'] = $int_links;
      $content['ext_links'] = $ext_links;
          
      return $content ;
    }
  }
  </code></pre>
  <p>So whats happening is the function loads the HTML via DomDocument
  I create 2 arrays to store both internal &amp; external</p>
  <p>Loop through the document and getElementsByTagName('a')</p>
  <p>It then uses strpos to check if the host address &quot;example.com&quot; is within the link URL if there is NO match/false then it's external, but we do a further check to make sure the link URL doesnt start with a forward slash ie: &quot;<strong>/contact-us.php</strong>&quot; that would mean its an internal, also in that check we check for a &quot;<strong>#</strong>&quot; tag at the begining which would be an anchor link on the page...</p>
  <p>so that was IF pos === false / no match
  now If host is in the link URL its a match, I do another check to see if the position of the host is lower down in the string, which would be internal ie:
  <a href="http://example.com/about/" rel="nofollow noreferrer">http://example.com/about/</a></p>
  <p>but if the position is greater than 20 (just a number plucked from the air) then..
  like a google plus link or facebook link the host url will be present in the link but much further along the string which would mean its an external,</p>
  <p>ie: <a href="http://www.facebook.com/plugins/like.php?href=http://example.com/" rel="nofollow noreferrer">http://www.facebook.com/plugins/like.php?href=http://example.com/</a></p>
  <p>phew...</p>
  <p>if you guys have any other BETTER ways to spot an external or internal link please let me know..my results, really vary depending on the site, if links are using the full path,</p>