title: Bash: Process monitoring and manipulation
body: <p>I have a C program that processes some input file. I'm using a Bash script to feed the input files one-by-one to this program, along with some other parameters. Each input file is processed by the program 4 times , each time by varying some parameters. You can think of it as an experiment to test the C program with different parameters.</p>
  
  <p>This C program can consume memory very quickly (and can even take up more than 95% of the OS memory , resulting in slowing down the system). So, in my script, I'm monitoring 2 things for <em>every test run</em> of the program - The total running time, and the memory percentage consumed (obtained from <code>top</code> command) . When either of them <strong><em>first</em></strong> crosses a threshold, I kill the C program using <code>killall -q 0 processname</code> , and begin the next test run.</p>
  
  <p>This is how my script is structured:  </p>
  
  <pre><code> # run in background
  ./program file_input1 file_input2 param1 param2 &amp;
  
   # now monitor the process
   # monitor time     
   sleep 1
   ((seconds++))
   if [ $seconds -ge $timeout ]; then
       timedout=1
       break
   fi
  
   # monitor memory percentage used
   memused=`top -bn1 | grep \`pidof genpbddh2\` | awk '{print $10}' | cut -d'.' -f1`
  
   if [ $memused -ge $memorylimit ]; then
      overmemory=1
      break
   fi
  </code></pre>
  
  <p>This entire thing is run inside a loop which keeps generating new values for the paramaters to the C program.<br>
  When a program breaks out of the loop due to timeout or over memory limit usage, this command is executed:<br>
  <code>killall -q 0 program</code></p>
  
  <p><strong>The problem:</strong><br>
  My intention was , once the program is started in the background (1st line above), I can monitor it. Then go to the next run of the program. A sequential execution of test cases.<br>
  But, it seems all the <em>future runs</em> of the program have been schedule by the OS (Linux) for some reason. That is, if Test Run 1 is running, Test Runs 2,3,4..and so on are also scheduled somehow (without Run 1 having finished). At least, it seems that way from the below observation:<br>
  When I pressed Ctrl-C to end the script, it exited cleanly. But, new instances of the "program" are keeping on being created continuously. The script has ended, but the instances of the program are still being continuously started. I checked and made sure that the script has ended. Now , I wrote a script to infinitely check for instances of this program being created and kill it. And eventually, all the pre-scheduled instances of the program were killed and no more new ones were created. But it was all a lot of pain.</p>
  
  <ol>
  <li>Is this the correct way to externally monitor a program?  </li>
  <li>Any clues on why this problem is occuring, and how to fix it?</li>
  </ol>