Keep my DOMDocument/DOMXpath PHP Script from Hogging Memory?
<p>Made this script to crawl certain links on a forum and extract the username, post date, and post number. </p>

<p>It works great, the only problem is that it hogs memory and after about a half hour it slows down significantly. </p>

<p>Does anyone have suggestions to speed it up? I've been running a WGET on my server to start the script. </p>

<p>Thanks,
Nick</p>

<pre><code>   &lt;?
//this php script is going to download pages and tear them apart from ###

/*
Here's the process:

1. prepare url 
2. get new HTML document from the web
3. extract xpath data
4. input in mysql database
*/


$baseURL="http://www.###.com";

//end viewtopic.php?p=357850
for ($post = 325479; $post &lt;= 357850; $post++) {

//connect to mysql
if (!mysql_connect('localhost','###','###')) echo mysql_error;
mysql_select_db('###');

//check to see if the post is already indexed
$result = mysql_query("SELECT postnumber FROM ### WHERE postnumber = '$post'");
if (mysql_num_rows($result) &gt; 0) {
    //echo "Already in the database." . "&lt;br&gt;";
    mysql_close();
    continue;
}

$url=$baseURL."/viewtopic.php?p=".$post;
//echo $url."&lt;br&gt;";

//get new HTML document
$html = new DOMDocument(); 
$html-&gt;loadHTMLFile($url);

$xpath = new DOMXpath($html);

//select the page elements that you want
//I want the parent of the TD class = forumRow
$links = $xpath-&gt;query( "//td[@class='forumRow']/parent::tr" ); 

    foreach($links as $results){
        $newDom = new DOMDocument;
        $newDom-&gt;appendChild($newDom-&gt;importNode($results,true));

        $xpath = new DOMXpath ($newDom);

        //which parts of the selection do you want?
        $time_stamp = trim($xpath-&gt;query("//td[2]/table/tr/td/span")-&gt;item(0)-&gt;nodeValue);
        $user_name = trim($xpath-&gt;query("//a[@class='genmed']")-&gt;item(0)-&gt;nodeValue);
        $post_number = trim($xpath-&gt;query("//td/a/@name")-&gt;item(0)-&gt;nodeValue);

        $return[] = array(
            'time_stamp' =&gt; $time_stamp,
            'username' =&gt; $user_name,
            'post_number' =&gt; $post_number,
            );
    }

    foreach ($return as $output) {
        if (strlen($output['time_stamp']) &gt; 0 &amp;&amp; strlen($output['username']) &gt; 0) 
          {
          //$timestamp = substr($output['time_stamp'],8,25);
          //echo $timestamp . "&lt;br&gt;";
          //$unixtimestamp = strtotime($timestamp);
          //echo $unixtimestamp;
          //echo $output['time_stamp']."&lt;br&gt;";
          preg_match("/[Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec]{3} \d{1,2}[,] \d{4} \d{1,2}[:]\d{2}/", $output['time_stamp'],$matches). "&lt;br&gt;";
          $unixtimestamp = strtotime($matches[0]);

          //YYYY-MM-DD HH:MM:SS
          $phpdate=date("Y-m-d H:i:s",$unixtimestamp);
          $username=$output['username'];
          $post_number=$output['post_number'];
          //echo $phpdate ." by ". $username . " #" . $post_number ;

          $result = mysql_query("SELECT postnumber FROM ### WHERE postnumber = '$post_number'");
          if (mysql_num_rows($result) == 0) {         
            if (mysql_query("INSERT INTO ### VALUES('','$url','$username','$phpdate','$post_number')")) echo "Y";
            else echo "N";
            mysql_close();
          }
          echo "&lt;br&gt;";
          }
    }
}
?&gt;
</code></pre>
