title: CakePHP Web Crawler Memory Leak
body: <p>I am developing an application that is, to put it simply, a niche based search engine. Within the application I have include a function crawl() which crawls a website and then uses the collectData() function to store the correct data from the site in the "products" table as described in the function. The visited pages are stored in a database</p>
  
  <p>The crawler works pretty well, just as described except for two things: Timeout and Memory. I've managed to correct the timeout error but the memory remains. I know simply increasing the memory_limit is not actually fixing the problem.</p>
  
  <p>The function is run by visiting "EXAMPLE.COM/products/crawl".</p>
  
  <p>Is a memory leak inevitable with a PHP Web crawler? OR is there something I'm doing wrong/not doing.</p>
  
  <p>Thanks in advance. (CODE BELOW)</p>
  
  <pre><code>function crawl() {
  
          $this-&gt;_crawl('http://www.example.com/','http://www.example.com');  
      }
  
      /***
      *
      * This function finds all link in $start and collects 
      * data from them as well as recursively crawling them
      *
      * @ param $start, the webpage where the crawler starts
      *
      * @ param $domain, the domain in which to stay
      *
      ***/
  
      function _crawl($start, $domain) {
          $dom = new DOMDocument();
          @$dom-&gt;loadHTMLFile($start);
  
          $xpath = new DOMXPath($dom);
          $hrefs = $xpath-&gt;evaluate("/html/body//a");//get all &lt;a&gt; elements
  
          for ($i = 0; $i &lt; $hrefs-&gt;length; $i++) {
  
              $href = $hrefs-&gt;item($i);
              $url = $href-&gt;getAttribute('href'); // get href value
              if(!(strpos($url, 'http') !== false)) {  //check for relative links
                  $url = $domain . '/' . $url;
              }
  
              if($this-&gt;Page-&gt;find('count', array('conditions' =&gt; array('Page.url' =&gt; $url))) &lt; 1 &amp;&amp; (strpos($url, $domain) !== false)) { // if this link has not already been crawled ( exists in database)
  
                  $this-&gt;Page-&gt;create();
                  $this-&gt;Page-&gt;set('url',$url);
                  $this-&gt;Page-&gt;set('indexed',date('Y-m-d H:i:s'));
                  $this-&gt;Page-&gt;save(); // add this url to database
  
                  $this-&gt;_collectData($url); //collect this links data
                  $this-&gt;_crawl($url, $domain); //crawl this link
              }
          }
      }
  </code></pre>