Scrape Page & Separate Internal & external links
<p>Building a little PHP scrapper, I'm writing a little function which should separate my Internal &amp; External links, </p>

<p>I'm passing the function a copy of the html source code along with the base host address</p>

<pre><code>$source = file_get_contents('http://www.mysite.com');
$host   = "mysite.com";
</code></pre>

<p>here is my function so far...</p>

<pre><code>function find_page_links($source, $host){

    if($source){

    $htmlDoc = new DomDocument();
    @$htmlDoc-&gt;loadhtml($source);

    $int_links = array();
    $ext_links = array();

    // GET LINKS
    foreach($htmlDoc-&gt;getElementsByTagName('a') as $link) {

        $url   = trim($link-&gt;getAttribute('href'));
        $title = trim($link-&gt;getAttribute('title'));
        $text  = trim($link-&gt;nodeValue);
        $rel   = trim($link-&gt;getAttribute('rel'));

        $pos = strpos($url,$host);

        if( $pos === false ){ // NO MATCH EXTERNAL              
            if( (substr($url, 0, 1) == '/') || 
                        (substr($url, 0, 1) == '#') )
                    {
            // INTERNAL
            $int_links[] = array( 'link_url' =&gt; $url, 
                                          'link_text' =&gt; $text, 
                                          'link_title' =&gt; $title, 
                                          'link_rel' =&gt; $rel 
                                        );
            }else{
            // EXTERNAL
            $ext_links[] = array( 'link_url' =&gt; $url, 
                                          'link_text' =&gt; $text, 
                                          'link_title' =&gt; $title, 
                                          'link_rel' =&gt; $rel 
                                        );
            }
        }else{
             if( $pos &lt; 20 ){
             // INTERNAL
             $int_links[] = array( 'link_url' =&gt; $url, 
                                       'link_text' =&gt; $text, 
                                       'link_title' =&gt; $title, 
                                       'link_rel' =&gt; $rel );
             }else{
             // EXTERNAL
             $ext_links[] = array( 'link_url' =&gt; $url, 
                                         'link_text' =&gt; $text, 
                                         'link_title' =&gt; $title, 
                                         'link_rel' =&gt; $rel 
                                        );  
             }
        } // end else
    } // end foreach

    $content = array();
    $content['int_links'] = $int_links;
    $content['ext_links'] = $ext_links;

    return $content ;
  }
}
</code></pre>

<p>So whats happening is the function loads the HTML via DomDocument
I create 2 arrays to store both internal &amp; external</p>

<p>Loop through the document and getElementsByTagName('a')</p>

<p>It then uses strpos to check if the host address "mysite.com" is within the link URL if there is NO match/false then it's external, but we do a further check to make sure the link URL doesnt start with a forward slash ie: "<strong>/contact-us.php</strong>" that would mean its an internal, also in that check we check for a "<strong>#</strong>" tag at the begining which would be an anchor link on the page...</p>

<p>so that was IF pos === false / no match
now If host is in the link URL its a match, I do another check to see if the position of the host is lower down in the string, which would be internal ie:
<a href="http://mysite.com/about/" rel="nofollow">http://mysite.com/about/</a></p>

<p>but if the position is greater than 20 (just a number plucked from the air) then..
like a google plus link or facebook link the host url will be present in the link but much further along the string which would mean its an external, </p>

<p>ie: <a href="http://www.facebook.com/plugins/like.php?href=http://mysite.com/" rel="nofollow">http://www.facebook.com/plugins/like.php?href=http://mysite.com/</a></p>

<p>phew...</p>

<p>if you guys have any other BETTER ways to spot an external or internal link please let me know..my results, really vary depending on the site, if links are using the full path, </p>
