Run 'N' Number of PHP Scripts on shell forever in a loop My objective is to run some 'N' number of PHP Scripts in cron. 

Since these are web scrapers coded in PHP and hits database on my server, each script runs for like 1 hour and they take a lot of CPU and memory. That's why I want to run one script at a time.

My first approach was to run each script at specific hour of the day but then there are only 24 hours in a day, so I can't really go with that way.

I have prepared a Unix script which runs in an infinite loop and runs the entire script. Assuming the shell will execute one script, wait for some time and then execute the next script. This is my script 1, say.

Then there is another script, Script 2 which runs every 3 hours and checks if Script 1 is running. If Script 1 is not running it will execute Script 1.

My concerns:


Is it feasible to run like 100 PHP scripts from one Unix script? If one fails, will it fail the whole script?
Is there any better approach?


Script 1 (MasterCronCarlos.sh):

#!/bin/bash
# Description = Script to check cpu usage and if above threshold issue a linux based command.
# This Script only checks "user cpu time (us - userspace (doing userspace stuff))
#


SUBJECT="CRON Status | Jai Carlos"
TO="myemail@gmail.com"

while true
do

echo -en "Starting fresh loop of cron" | mail -s "$SUBJECT" "$TO"


    php /home/site1/public_html/scrapperM1.php
    sleep 300
    php /home/site1/public_html/scrapperM2.php
    sleep 300
    php /home/site1/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site1.com" | mail -s "$SUBJECT" "$TO"

    php /home/site2/public_html/scrapperM1.php
    sleep 300
    php /home/site2/public_html/scrapperM2.php
    sleep 300
    php /home/site2/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site2.tv" | mail -s "$SUBJECT" "$TO"

    php /home/site3/public_html/scrapperM1.php
    sleep 300
    php /home/site3/public_html/scrapperM2.php
    sleep 300
    php /home/site3/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site3" | mail -s "$SUBJECT" "$TO"

    php /home/site4/public_html/scrapperM1.php
    sleep 300
    php /home/site4/public_html/scrapperM2.php
    sleep 300
    php /home/site4/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site4" | mail -s "$SUBJECT" "$TO"

    php /home/site5/public_html/scrapperM1.php
    sleep 300
    php /home/site5/public_html/scrapperM2.php
    sleep 300
    php /home/site5/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site5" | mail -s "$SUBJECT" "$TO"

    php /home/site6/public_html/scrapperM1.php
    sleep 300
    php /home/site6/public_html/scrapperM2.php
    sleep 300
    php /home/site6/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site6" | mail -s "$SUBJECT" "$TO"

    php /home/site7/public_html/scrapperM1.php
    sleep 300
    php /home/site7/public_html/scrapperM2.php
    sleep 300
    php /home/site7/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site7" | mail -s "$SUBJECT" "$TO"

done


Script 2 (mastercron.sh):

#!/bin/sh

RESULT=`ps axf | grep "MasterCronCarlos.sh" | grep -v grep | awk '{print  $1}'`

if [ $RESULT -ge 0 ]; then

echo "Running"

else

echo "Not Running"

sh /home/backups/MasterCronCarlos.sh

fi;
I think more than even a code review, what you are really needing to think about is your architecture.  In the modern age of event-based workflows and parallel computing should you really be managing and executing a complex set of scripts in this manner (i.e. via shell automation)?

My advice is to look for better tooling that will allow you to orchestrate your pieces of logic via an automated workflow of some sort.  Even consider things like map reduce tools (Hadoop, Elastic Map Reduce, etc.) which are ideal for parallel workload execution with aggregation of results.