How to execute a script that takes a long time? I am executing a script by calling a URL using the wget command. I want to execute this script 3 times a day for 2 hours each time. To do so, I check the difference between the start time and the current time. 

My problem is that sometimes the script runs for 2 hours, sometime 1 hour, and sometimes just 3 minutes. I don't know why. I have increased memory_limit and max_execution_time. Still, I haven't been able to find the reason.

What my script is doing exactly:


It fetches a data from other site using cUrl. The data are articles, journals and some other user's data.
After getting the data in XML format, it adds each content.
You can break up your work somehow. You shouldn't try to process thousands of things at a time and assume nothing will break or you'll run out of time.

I have recently wrote an in-house Drupal 7 project that allows me to query a webservice for XML data very quickly and process 100,000s of XML entries to an SQL DB pretty fast. A full fetch of all remote SOAP based data is ~100,000 SQL rows after import. This takes ~5 mins to complete. From CURL to DB insert.

The Drupal 7 site performs the following:


Allows me to fetch a .NET SOAP response for X departments from my university. X any number of department queries from 1 department to 90 departments per school quarter.

I used the Background Process module to fetch the CURL requests to local XML files in a Drupal private filesystem.
90 CURL requests with ~2Mb XML responses complete in ~1 minute. (I could request 1 huge xml file of all departments; I break it up for many threads below).
The XML files are added to a Drupal queue. Queue's can lease queue items as jobs for processing. A developer can set a lease time per job for any length of time.

I then use hook_cron_queue to define a worker function that parses the XML and inserts the data into a database. This happens to be a non-Drupal database that I connect to via a custom db entry in settings.php.

I use the Ultimate Cron and Ultimate Cron Queue Scaler modules to actually perform the processing. I set 9 threads in Ultimate Cron Queue to process queue items, and I have set my lease time to literally 2 hours (7600 seconds). Drupal cron runs/checks for work every minute, the lease time assures no thread tramples over work already in progress.

with no optimizations a script I'd written using PHP DomDocument with Xpath queries inserting SQL queries to a DB took minutes per XML file. After adding some SQL batch inserts with db_insert each file takes seconds, but I know from progressively optimizing my scripts a job can have a set lease time. Which is what you want here.




In short. Find a way to break-up your workload and dont be afraid to use watchdog log messages in your module code. A Queue is a nice way to store pieces of the whole task to complete in tiny parts. As you run your worker functions you'll be able to find a lease time that works best for your data size and processing times.

I realized your post is about Drupal 6; it doesn't really matter. The core pieces you want are the D6 queue project and hook_cron(). The Ultimate Cron, Ultimate Cron Queue Scaler, and the Background Process modules have D6 versions.

The project described is a private in-house project, and I cannot easily provide code examples. I've tried to outline how to glue this all together.