Php run time forever and Web Parsers
<p>so i found a nice slice of code, well its best to say a theoretical example of what might be the code for a decent web parse. 
I recall that when i got into this issue myself a while back i could not get the script to run for an infinite time span in fact i couldn't have it run more then a few hours. </p>

<p>This is after i set the following:</p>

<pre><code>    set_time_limit(0);
    ini_set("memory_limit", "800M");
// in some case i would set them both in others, just one or the other
</code></pre>

<p>I have been doing some reading as to how to get PHP to run for a longer time span, an infinity stretch to be exact.
I found many suggestions for cron job instead of PHP yet i would like to find a way to have this done in php. </p>

<p>I would love some examples, even theoretical if thats all you can muster. </p>

<p>I would like to use this block of code i mentioned above, which i found here as a reference to get the discussion started on the right path. </p>

<p><a href="http://stackoverflow.com/questions/102631/how-to-write-a-crawler">How to write a crawler?</a>   </p>

<pre><code>  while(list of unvisited URLs is not empty) {
     take URL from list
     fetch content
     record whatever it is you want to about the content
     if content is HTML {
     parse out URLs from links
     foreach URL {
        if it matches your rules
          and it's not already in either the visited or unvisited list
          add it to the unvisited list
     }
   }
 }
</code></pre>
