Concurrency with shell scripts in failure-prone environments
Question: Good morning all,

I am trying to implement concurrency in a very specific environment, and keep
getting stuck. Maybe you can help me.

this is the situation:

-I have N nodes that can read/write in a shared folder. 

-I want to execute an application in one of them. this can be anything, like a shell script, an installed code, or whatever. 

-To do so, I have to send the same command to all of them. The first one should start the execution, and the rest should see that somebody else is running the desired application and exit.

-The execution of the application can be killed at any time. This is important because does not allow relying on any cleaning step after the execution. 

-if the application gets killed, the user may want to execute it again. He would then send the very same command. 

My current approach is to create a shell script that wraps the command to be
executed. This could also be implemented in C. Not python or other languages,
to avoid library dependencies.

    
    
    #!/bin/sh
    # (folder structure simplified for legibility)
    mutex(){
      lockdir=".lock"
      firstTask=1 #false
    
      if mkdir "$lockdir"  &> /dev/null
      then
        controlFile="controlFile"
    
        #if this is the first node, start coordinator
        if [ ! -f $controlFile ]; then
          firstTask=0 #true
          #tell the rest of nodes that I am in control
          echo "some info" > $controlFile
         fi
    
         # remove control File when script finishes
         trap 'rm $controlFile' EXIT
      fi
      return $firstTask
    }
    
    
    #The basic idea is that a task executes the desire command, stated as arguments to this script. The rest do nothing
    if  ! mutex ;
    then
      exit 0
    fi
    
    #I am the first node and the only one reaching this, so I execute whatever
    $@
    

If there are no failures, this wrapper works great. The problem is that, if
the script is killed before the execution, the trap is not executed and the
control file is not removed. Then, when we execute the wrapper again to
restart the task, it won't work as every node will think that somebody else is
running the application.

A possible solution would be to remove the control script just before the "$@"
call, but that it would lead to some race condition.

Any suggestion or idea?

Thanks for your help.

**edit** : edited with correct solution as future reference


Answer: Your trap syntax looks wrong: According to POSIX, it should be:

    
    
    trap [action condition ...]
    

e.g.:

    
    
    trap 'rm $controlFile' HUP INT TERM
    trap 'rm $controlFile' 1 2 15
    

Note that `$controlFile` will not be expanded until the trap is executed if
you use single quotes.

