title: Crawl a website, get the links, crawl the links with PHP and XPATH
body: <p>I want to crawl an entire website , I have read several threads but I cannot manage to get data in a 2nd level.</p>
  
  <p>That is, I can return the links from a starting page but then I cannot find a way to parse the links and get the content of each link...</p>
  
  <p>The code I use is:</p>
  
  <pre><code>&lt;?php
  
      //  SELECT STARTING PAGE
        $url = 'http://mydomain.com/';
        $html= file_get_contents($url);
  
       // GET ALL THE LINKS OF EACH PAGE
  
           // create a dom object
  
              $dom = new DOMDocument();
              @$dom-&gt;loadHTML($html);
  
           // run xpath for the dom
  
              $xPath = new DOMXPath($dom);
  
  
           // get links from starting page
  
              $elements = $xPath-&gt;query("//a/@href");
              foreach ($elements as $e) {
              echo $e-&gt;nodeValue. "&lt;br /&gt;";
              }
  
       // Parse each page using the extracted links?
  
   ?&gt;
  </code></pre>
  
  <p>Could somebody help me out for the last part with an example?</p>
  
  <p>I will be really much appreciated!</p>
  
  <hr>
  
  <p>Well , thanx for your answers!
  I tried some stuff but I Haven't managet to get any results yet - I am new to programming..</p>
  
  <p>Below, you can find 2 of my attempts -  the 1st trying to parse the links  and in the second trying to replace file_get contents with Curl:</p>
  
  <pre><code> 1) 
  
  &lt;?php 
    //  GET STARTING PAGE
    $url = 'http://www.capoeira.com.gr/';
    $html= file_get_contents($url);
  
    //GET ALL THE LINKS FROM STARTING PAGE
  
    // create a dom object
  
      $dom = new DOMDocument();
      @$dom-&gt;loadHTML($html);
  
  
      // run xpath for the dom
  
      $xPath = new DOMXPath($dom);
  
          // get specific elements from the sites
  
          $elements = $xPath-&gt;query("//a/@href");
  //PARSE EACH LINK
  
      foreach($elements as $e) {
            $URLS= file_get_contents($e);
            $dom = new DOMDocument();
            @$dom-&gt;loadHTML($html);
            $xPath = new DOMXPath($dom);
            $output = $xPath-&gt;query("//div[@class='content-entry clearfix']");
           echo $output -&gt;nodeValue;
          }                           
           ?&gt;
  </code></pre>
  
  <p>For the above code I get 
  Warning: file_get_contents() expects parameter 1 to be string, object given in ../example.php on line 26</p>
  
  <p>2)<br>
               
  
  <pre><code>    &lt;?php
            $curl = curl_init();
            curl_setopt($curl, CURLOPT_POST, 1);
            curl_setopt($curl, CURLOPT_URL, "http://capoeira.com.gr");
            curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
            $content= curl_exec($curl);
            curl_close($curl);    
  
  
  
            $dom = new DOMDocument();
            @$dom-&gt;loadHTML($content);
  
             $xPath = new DOMXPath($dom);
             $elements = $xPath-&gt;query("//a/@href");
              foreach ($elements as $e) {
              echo $e-&gt;nodeValue. "&lt;br /&gt;";
              }
  
     ?&gt;
  </code></pre>
  
  <p>I get no results. I tried to echo $content and then I get :</p>
  
  <p>You don't have permission to access / on this server.</p>
  
  <p>Additionally, a 413 Request Entity Too Large error was encountered while trying to use an ErrorDocument to handle the request...</p>
  
  <p>Any ideas please?? :)</p>