How can I more efficiently save nodes programatically with my custom module without cron? I have a scraper I created that aggregates data found on various sites, feeds, and databases. So that I don't hammer the servers to death that I am connecting to I have it set to sleep 5 seconds between requests.  When it finds what I need it saves it as a new node.  I have program set to fire on a cron run using hook_cron().

It's works fine but now I have added significantly to the sources I am searching through. The problem is this process could days, especially doing it in series. I have even had the case be that I get an error that cron tried to run while cron was already running.

How can I make this process go more efficiently? More quickly? And not depend on cron? Before I wrote a module for this I simply created several scripts - one script for each source -  and I fired them off simultaneously in terminal windows.

How can I do this with my module? I know I could run php's exec() function. In fact this is the commanad was calling with it when I was doing it manually:

php myscripts/search1.php &lt; /dev/null &amp;
php myscripts/search2.php &lt; /dev/null &amp;


When I try this in the module I get bootstrap errors.

When I wrote the module my individual scripts simply became functions in the module. I'm happy to go back to firing off the scripts simultaneously if someone can help me understand how to do that correctly in the module. Or, if you have another suggestion on how to do this quickly and efficiently, I'm all ears.  Realistically, I need to process a couple million items of content a week.
To fix the bootstrap issue you need to change the CWD (Current Working Directory) to where index.php is located. So in your case you are one directory deep so adding this to the top of your scripts inside of "myscripts" should make them work

// Change the directory to the Drupal root.
chdir('..');


If you were 2 directories deep the code would look like this chdir('../..');. If the errors had to do with php notices/warnings and things of that nature I would look at scripts/drupal.sh (checkout the "define default settings" section) or running your commands/functions with Drush.

There are a couple of options to help with dealing with millions of items of content per week. My recommendation is the HTTP Parallel Request &amp; Threading Library. It will help with downloading things in parallel and doing things in background processes. As an example Link Checker recently integrated with it http://drupal.org/node/380052. With HTTPRL, Link Checker runs _linkchecker_check_links in a new process independent of cron, issue parallel http requests, and runs a callback inside of the event loop allowing work to be done locally while we wait for things to download.

The other options (That I know of) for dealing with LOTS of items are Elysia Cron and Background Process. The stock Batch API will be too slow if your looking to do millions of things per week (there are only 604,800 seconds in a week).