Website monitor - if site is down, kill the process, contact admin and restart server My Python-based web server (Tornado module) is pretty complex and keeps going down (seems like usually in the middle of the night), so here's a bash script that kills the process, restarts and emails and txt messages me:

#!/bin/bash
NOTIFYEMAIL=&lt;my_email_address&gt;
SMSEMAIL=&lt;cell_phone_number&gt;@&lt;carrier&gt; # ie. pm.sprint.com
SENDEREMAIL=admin@loclahost
SERVER=http://the_website.com/
PAUSE=60
FAILED=0
DEBUG=0

while true 
do
/usr/bin/curl -sSf --max-time 10 $SERVER &gt; /dev/null 2&gt;&amp;1
CS=$?
# For debugging purposes
if [ $DEBUG -eq 1 ]
then
    echo "STATUS = $CS"
    echo "FAILED = $FAILED"
    if [ $CS -ne 0 ]
    then
    echo "$SERVER is down"

    elif [ $CS -eq 0 ]
    then
    echo "$SERVER is up"
    fi
fi

# If the server is down and no alert is sent - alert
if [ $CS -ne 0 ] &amp;&amp; [ $FAILED -eq 0 ]
then
    PID=`ps -eaf | grep 'python -m fore.server' | grep -v grep | awk '{print $2}'`
    FAILED=1
    if [ $DEBUG -eq 1 ]
    then
    echo "$SERVER failed"
    if [[ "" !=  "$PID" ]]
    then
        echo "killing $PID"
        kill -9 $PID
        echo "Starting the server."
        python -m fore.server
    else
        echo "No Python Process Running Locally"
    fi
    fi
    if [ $DEBUG = 0 ]
    then
    echo "$SERVER failed"
    PID=`ps -eaf | grep 'python -m fore.server' | grep -v grep | awk '{print $2}'`
    if [[ "" !=  "$PID" ]]
    then
        kill -9 $PID
        echo "$SERVER went down $(date)" | /usr/bin/mail -s "$SERVER went down" "$SENDEREMAIL" "$SMSEMAIL" 
        echo "$SERVER went down $(date)" | /usr/bin/mail -s "$SERVER went down" "$SENDEREMAIL" "$NOTIFYEMAIL"
        python -m fore.server
    else
        echo "No Python Process Running Locally"
    fi 
    fi

# If the server is back up and no alert is sent - alert
elif [ $CS -eq 0 ] &amp;&amp; [ $FAILED -eq 1 ]
then
    FAILED=0
    if [ $DEBUG -eq 1 ]
    then
    echo "$SERVER is back up"
    fi
    if [ $DEBUG = 0 ]
    then
    echo "$SERVER is back up $(date)" | /usr/bin/mail -s "$SERVER is back up again" "$SENDEREMAIL" "$SMSEMAIL"
    echo "$SERVER is back up $(date)" | /usr/bin/mail -s "$SERVER is back up again" "$SENDEREMAIL" "$NOTIFYEMAIL"
    fi
fi
sleep $PAUSE
done


I have to execute my server with sudo in order to use port :80 so I am executing the script like:

sudo ./outage.sh


Of course, warnings, suggestions, improvements are welcome. 

One improvement I'm interested in implementing is to iterate through a list of servers in order to keep track of multiple web sites.
Indent code blocks

To improve readability,
it's good to indent code blocks in loops and if statements.
For example:

while true
do
    /usr/bin/curl -sSf --max-time 10 $SERVER &gt; /dev/null 2&gt;&amp;1
    CS=$?


You indented some if statements, but not all. Indent all.

Replace elif with else if possible

In these conditions:


    if [ $CS -ne 0 ]
    then
    echo "$SERVER is down"

    elif [ $CS -eq 0 ]
    then
    echo "$SERVER is up"



If $CS is not-not-zero, then when can it be? Zero, of course!
So the elif can be a simple else.

Use $(...) instead of `...`

`...` is old-fashioned and error prone. Use $(...) instead.

ps ... | grep ... | grep -v grep

Instead of:


ps -eaf | grep 'python -m fore.server' | grep -v grep



A common workaround is:


ps -eaf | grep '[p]ython -m fore.server'



This works, because this way in the output of ps, the process python -m fore.server will still match, but grep '[p]ython -m fore.server' will not match.

However, in this particular example you should go a bit further, see the next point.

Look suspiciously at grep in a pipeline ending with awk

awk is fast and powerful,
and very often can easily do grep's job too when it's in the same pipeline.

For example instead of this:


ps -eaf | grep 'python -m fore.server' | grep -v grep | awk '{print $2}'



You can rewrite the whole thing using awk alone:

ps -eaf | awk '/python -m fore.server/ {print $2}'


This is not only shorter and less troublesome (no need for the grep [p]ython trick),
and there are less processes executed in the pipeline.

Don't repeat yourself

You do some things twice, with identical or very similar commands:


Finding the PID of the Python process
Sending an email with the message that the server is down
Sending an email with the message that the server is back up


It would be better to move common operations to helper functions.

Other simplifications

Instead of this:


    if [[ "" !=  "$PID" ]]



You can write as:

    if [[ $PID ]]