title: How to start a large number of quick jobs in Bash
body: <p>I have 3000 very quick jobs to run that on average take 2/3 seconds.</p>
  
  <p>The list of jobs is in a file, and I want to control how many I have open.</p>
  
  <p>However, the process of starting a job in background (&amp; line) seems to take some time itself, therefore, some jobs are already finishing before "INTOTAL" amount have got started...</p>
  
  <p>Therefore, I am not using my 32 core efficiently. </p>
  
  <p>Is the a better approach than the one below?</p>
  
  <pre><code>#!/bin/sh
  #set -x
  
  INTOTAL=28
  
  
  while true
  do
  
      NUMRUNNING=`tasklist | egrep Prod.exe | wc -l`
  
      JOBS=`cat jobs.lst | wc -l`
      if [ $JOBS -gt 0 ]
      then
  
          MAXSTART=$(($INTOTAL-$NUMRUNNING))
          NUMTOSTART=$JOBS
  
          if [ $NUMTOSTART -gt $MAXSTART ]
          then
              NUMTOSTART=$MAXSTART
          fi
          echo 'Starting: '$NUMTOSTART 
          for ((i=1;i&lt;=$NUMTOSTART;i++))
          do
              JOB=`head -n1 jobs.lst`
              sed -i 1d jobs.lst
  
              /Prod $JOB &amp;
  
          done
          sleep 2
      fi
  
  
      sleep 3
  
  done
  </code></pre>