<p>Lets say I have a loop in Bash:</p>

<pre><code>for foo in `some-command`
do
   do-something $foo
done
</code></pre>

<p><code>do-something</code> is cpu bound and I have a nice shiny 4 core processor. I'd like to be able to run up to 4 <code>do-something</code>'s at once.</p>

<p>The naive approach seems to be:</p>

<pre><code>for foo in `some-command`
do
   do-something $foo &amp;
done
</code></pre>

<p>This will run <em>all</em> <code>do-something</code>s at once, but there are a couple downsides, mainly that do-something may also have some significant I/O which performing <em>all</em> at once might slow down a bit. The other problem is that this code block returns immediately, so no way to do other work when all the <code>do-something</code>s are finished.</p>

<p>How would you write this loop so there are always X <code>do-something</code>s running at once?</p>

[{'answer_id': 38166, 'author': 'skolima', 'author_id': 3205, 'author_profile': 'https://Stackoverflow.com/users/3205', 'pm_score': 4, 'selected': False, 'text': '<p>Instead of a plain bash, use a Makefile, then specify number of simultaneous jobs with <code>make -jX</code> where X is the number of jobs to run at once.</p>\n\n<p>Or you can use <code>wait</code> ("<code>man wait</code>"): launch several child processes, call <code>wait</code> - it will exit when the child processes finish.</p>\n\n<pre><code>maxjobs = 10\n\nforeach line in `cat file.txt` {\n jobsrunning = 0\n while jobsrunning &lt; maxjobs {\n  do job &amp;\n  jobsrunning += 1\n }\nwait\n}\n\njob ( ){\n...\n}\n</code></pre>\n\n<p>If you need to store the job\'s result, then assign their result to a variable. After <code>wait</code> you just check what the variable contains.</p>\n'}, {'answer_id': 38171, 'author': 'tessein', 'author_id': 3075, 'author_profile': 'https://Stackoverflow.com/users/3075', 'pm_score': 3, 'selected': False, 'text': '<p>Maybe try a parallelizing utility instead rewriting the loop? I\'m a big fan of xjobs. I use xjobs all the time to mass copy files across our network, usually when setting up a new database server.\n<a href="http://www.maier-komor.de/xjobs.html" rel="noreferrer">http://www.maier-komor.de/xjobs.html</a></p>\n'}, {'answer_id': 38225, 'author': 'bstark', 'author_id': 4056, 'author_profile': 'https://Stackoverflow.com/users/4056', 'pm_score': 4, 'selected': False, 'text': '<pre>\nmaxjobs=4\nparallelize () {\n        while [ $# -gt 0 ] ; do\n                jobcnt=(`jobs -p`)\n                if [ ${#jobcnt[@]} -lt $maxjobs ] ; then\n                        do-something $1 &\n                        shift  \n                else\n                        sleep 1\n                fi\n        done\n        wait\n}\n\nparallelize arg1 arg2 "5 args to third job" arg4 ...\n</pre>\n'}, {'answer_id': 42809, 'author': 'Jon Ericson', 'author_id': 1438, 'author_profile': 'https://Stackoverflow.com/users/1438', 'pm_score': 2, 'selected': False, 'text': "<p>The project I work on uses the <strong>wait</strong> command to control parallel shell (ksh actually) processes.  To address your concerns about IO, on a modern OS, it's possible parallel execution will actually increase efficiency.  If all processes are reading the same blocks on disk, only the first process will have to hit the physical hardware.  The other processes will often be able to retrieve the block from OS's disk cache in memory.  Obviously, reading from memory is several orders of magnitude quicker than reading from disk.  Also, the benefit requires no coding changes.</p>\n"}, {'answer_id': 880864, 'author': 'Grumbel', 'author_id': 28113, 'author_profile': 'https://Stackoverflow.com/users/28113', 'pm_score': 4, 'selected': False, 'text': "<p>Here an alternative solution that can be inserted into .bashrc and used for everyday one liner:</p>\n\n<pre><code>function pwait() {\n    while [ $(jobs -p | wc -l) -ge $1 ]; do\n        sleep 1\n    done\n}\n</code></pre>\n\n<p>To use it, all one has to do is put <code>&amp;</code> after the jobs and a pwait call, the parameter gives the number of parallel processes:</p>\n\n<pre><code>for i in *; do\n    do_something $i &amp;\n    pwait 10\ndone\n</code></pre>\n\n<p>It would be nicer to use <code>wait</code> instead of busy waiting on the output of <code>jobs -p</code>, but there doesn't seem to be an obvious solution to wait till any of the given jobs is finished instead of a all of them.</p>\n"}, {'answer_id': 881392, 'author': 'lhunath', 'author_id': 58803, 'author_profile': 'https://Stackoverflow.com/users/58803', 'pm_score': 3, 'selected': False, 'text': '<p>While doing this right in <code>bash</code> is probably impossible, you can do a semi-right fairly easily.  <code>bstark</code> gave a fair approximation of right but his has the following flaws:</p>\n\n<ul>\n<li>Word splitting: You can\'t pass any jobs to it that use any of the following characters in their arguments: spaces, tabs, newlines, stars, question marks.  If you do, things will break, possibly unexpectedly.</li>\n<li>It relies on the rest of your script to not background anything.  If you do, or later you add something to the script that gets sent in the background because you forgot you weren\'t allowed to use backgrounded jobs because of his snippet, things will break.</li>\n</ul>\n\n<p>Another approximation which doesn\'t have these flaws is the following:</p>\n\n<pre><code>scheduleAll() {\n    local job i=0 max=4 pids=()\n\n    for job; do\n        (( ++i % max == 0 )) &amp;&amp; {\n            wait "${pids[@]}"\n            pids=()\n        }\n\n        bash -c "$job" &amp; pids+=("$!")\n    done\n\n    wait "${pids[@]}"\n}\n</code></pre>\n\n<p>Note that this one is easily adaptable to also check the exit code of each job as it ends so you can warn the user if a job fails or set an exit code for <code>scheduleAll</code> according to the amount of jobs that failed, or something.</p>\n\n<p>The problem with this code is just that:</p>\n\n<ul>\n<li>It schedules four (in this case) jobs at a time and then waits for all four to end.  Some might be done sooner than others which will cause the next batch of four jobs to wait until the longest of the previous batch is done.</li>\n</ul>\n\n<p>A solution that takes care of this last issue would have to use <code>kill -0</code> to poll whether any of the processes have disappeared instead of the <code>wait</code> and schedule the next job.  However, that introduces a small new problem: you have a race condition between a job ending, and the <code>kill -0</code> checking whether it\'s ended.  If the job ended and another process on your system starts up at the same time, taking a random PID which happens to be that of the job that just finished, the <code>kill -0</code> won\'t notice your job having finished and things will break again.</p>\n\n<p>A perfect solution isn\'t possible in <code>bash</code>.</p>\n'}, {'answer_id': 881450, 'author': 'Fritz G. Mehner', 'author_id': 57457, 'author_profile': 'https://Stackoverflow.com/users/57457', 'pm_score': 7, 'selected': True, 'text': '<p>Depending on what you want to do xargs also can help (here: converting documents with pdf2ps):</p>\n\n<pre><code>cpus=$( ls -d /sys/devices/system/cpu/cpu[[:digit:]]* | wc -w )\n\nfind . -name \\*.pdf | xargs --max-args=1 --max-procs=$cpus  pdf2ps\n</code></pre>\n\n<p>From the docs:</p>\n\n<pre><code>--max-procs=max-procs\n-P max-procs\n       Run up to max-procs processes at a time; the default is 1.\n       If max-procs is 0, xargs will run as many processes as  possible  at  a\n       time.  Use the -n option with -P; otherwise chances are that only one\n       exec will be done.\n</code></pre>\n'}, {'answer_id': 894986, 'author': 'Idelic', 'author_id': 109744, 'author_profile': 'https://Stackoverflow.com/users/109744', 'pm_score': 3, 'selected': False, 'text': "<p>If you're familiar with the <code>make</code> command, most of the time you can express the list of commands you want to run as a a makefile. For example, if you need to run $SOME_COMMAND on files *.input each of which produces *.output, you can use the makefile</p>\n\n<pre>\nINPUT  = a.input b.input\nOUTPUT = $(INPUT:.input=.output)\n\n%.output : %.input\n    $(SOME_COMMAND) $&lt; $@\n\nall: $(OUTPUT)\n</pre>\n\n<p>and then just run</p>\n\n<pre>\nmake -j&lt;NUMBER&gt;\n</pre>\n\n<p>to run at most NUMBER commands in parallel.</p>\n"}, {'answer_id': 3011097, 'author': 'Ole Tange', 'author_id': 363028, 'author_profile': 'https://Stackoverflow.com/users/363028', 'pm_score': 5, 'selected': False, 'text': '<p>With GNU Parallel <a href="http://www.gnu.org/software/parallel/" rel="noreferrer">http://www.gnu.org/software/parallel/</a> you can write:</p>\n\n<pre><code>some-command | parallel do-something\n</code></pre>\n\n<p>GNU Parallel also supports running jobs on remote computers. This will run one per CPU core on the remote computers - even if they have different number of cores:</p>\n\n<pre><code>some-command | parallel -S server1,server2 do-something\n</code></pre>\n\n<p>A more advanced example: Here we list of files that we want my_script to run on. Files have extension (maybe .jpeg). We want the output of my_script to be put next to the files in basename.out (e.g. foo.jpeg -> foo.out). We want to run my_script once for each core the computer has and we want to run it on the local computer, too. For the remote computers we want the file to be processed transferred to the given computer. When my_script finishes, we want foo.out transferred back and we then want foo.jpeg and foo.out removed from the remote computer:</p>\n\n<pre><code>cat list_of_files | \\\nparallel --trc {.}.out -S server1,server2,: \\\n"my_script {} &gt; {.}.out"\n</code></pre>\n\n<p>GNU Parallel makes sure the output from each job does not mix, so you can use the output as input for another program:</p>\n\n<pre><code>some-command | parallel do-something | postprocess\n</code></pre>\n\n<p>See the videos for more examples: <a href="https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1" rel="noreferrer">https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1</a></p>\n'}, {'answer_id': 6773673, 'author': 'cat', 'author_id': 712124, 'author_profile': 'https://Stackoverflow.com/users/712124', 'pm_score': 1, 'selected': False, 'text': '<p>This might be good enough for most purposes, but is not optimal.</p>\n\n<pre><code>#!/bin/bash\n\nn=0\nmaxjobs=10\n\nfor i in *.m4a ; do\n    # ( DO SOMETHING ) &amp;\n\n    # limit jobs\n    if (( $(($((++n)) % $maxjobs)) == 0 )) ; then\n        wait # wait until all have finished (not optimal, but most times good enough)\n        echo $n wait\n    fi\ndone\n</code></pre>\n'}, {'answer_id': 8196788, 'author': 'Adam Zalcman', 'author_id': 1023815, 'author_profile': 'https://Stackoverflow.com/users/1023815', 'pm_score': 0, 'selected': False, 'text': '<p>You can use a simple nested for loop (substitute appropriate integers for N and M below):</p>\n\n<pre><code>for i in {1..N}; do\n  (for j in {1..M}; do do_something; done &amp; );\ndone\n</code></pre>\n\n<p>This will execute do_something N*M times in M rounds, each round executing N jobs in parallel. You can make N equal the number of CPUs you have.</p>\n'}, {'answer_id': 9392829, 'author': 'ilnar', 'author_id': 1225511, 'author_profile': 'https://Stackoverflow.com/users/1225511', 'pm_score': 2, 'selected': False, 'text': '<p>function for bash:</p>\n\n<pre><code>parallel ()\n{\n    awk "BEGIN{print \\"all: ALL_TARGETS\\\\n\\"}{print \\"TARGET_\\"NR\\":\\\\n\\\\t@-\\"\\$0\\"\\\\n\\"}END{printf \\"ALL_TARGETS:\\";for(i=1;i&lt;=NR;i++){printf \\" TARGET_%d\\",i};print\\"\\\\n\\"}" | make $@ -f - all\n}\n</code></pre>\n\n<p>using:</p>\n\n<pre><code>cat my_commands | parallel -j 4\n</code></pre>\n'}, {'answer_id': 21156538, 'author': 'Jack', 'author_id': 1327414, 'author_profile': 'https://Stackoverflow.com/users/1327414', 'pm_score': -1, 'selected': False, 'text': '<p>$DOMAINS = "list of some domain in commands"\nfor foo in <code>some-command</code>\ndo</p>\n\n<pre><code>eval `some-command for $DOMAINS` &amp;\n\n    job[$i]=$!\n\n    i=$(( i + 1))\n</code></pre>\n\n<p>done</p>\n\n<p>Ndomains=<code>echo $DOMAINS |wc -w</code></p>\n\n<p>for i in $(seq 1 1 $Ndomains)\ndo\n  echo "wait for ${job[$i]}"\n  wait "${job[$i]}"\ndone</p>\n\n<p>in this concept will work for the parallelize. important thing is last line of eval is \'&amp;\'\nwhich will put the commands to backgrounds.</p>\n'}, {'answer_id': 33108323, 'author': 'Fernando', 'author_id': 1543263, 'author_profile': 'https://Stackoverflow.com/users/1543263', 'pm_score': 1, 'selected': False, 'text': '<p>Here is how I managed to solve this issue in a bash script:</p>\n\n<pre><code> #! /bin/bash\n\n MAX_JOBS=32\n\n FILE_LIST=($(cat ${1}))\n\n echo Length ${#FILE_LIST[@]}\n\n for ((INDEX=0; INDEX &lt; ${#FILE_LIST[@]}; INDEX=$((${INDEX}+${MAX_JOBS})) ));\n do\n     JOBS_RUNNING=0\n     while ((JOBS_RUNNING &lt; MAX_JOBS))\n     do\n         I=$((${INDEX}+${JOBS_RUNNING}))\n         FILE=${FILE_LIST[${I}]}\n         if [ "$FILE" != "" ];then\n             echo $JOBS_RUNNING $FILE\n             ./M22Checker ${FILE} &amp;\n         else\n             echo $JOBS_RUNNING NULL &amp;\n         fi\n         JOBS_RUNNING=$((JOBS_RUNNING+1))\n     done\n     wait\n done\n</code></pre>\n'}, {'answer_id': 39189370, 'author': 'Orsiris de Jong', 'author_id': 2635443, 'author_profile': 'https://Stackoverflow.com/users/2635443', 'pm_score': 0, 'selected': False, 'text': '<p>My solution to always keep a given number of processes running, keep tracking of errors and handle ubnterruptible / zombie processes:</p>\n\n<pre><code>function log {\n    echo "$1"\n}\n\n# Take a list of commands to run, runs them sequentially with numberOfProcesses commands simultaneously runs\n# Returns the number of non zero exit codes from commands\nfunction ParallelExec {\n    local numberOfProcesses="${1}" # Number of simultaneous commands to run\n    local commandsArg="${2}" # Semi-colon separated list of commands\n\n    local pid\n    local runningPids=0\n    local counter=0\n    local commandsArray\n    local pidsArray\n    local newPidsArray\n    local retval\n    local retvalAll=0\n    local pidState\n    local commandsArrayPid\n\n    IFS=\';\' read -r -a commandsArray &lt;&lt;&lt; "$commandsArg"\n\n    log "Runnning ${#commandsArray[@]} commands in $numberOfProcesses simultaneous processes."\n\n    while [ $counter -lt "${#commandsArray[@]}" ] || [ ${#pidsArray[@]} -gt 0 ]; do\n\n        while [ $counter -lt "${#commandsArray[@]}" ] &amp;&amp; [ ${#pidsArray[@]} -lt $numberOfProcesses ]; do\n            log "Running command [${commandsArray[$counter]}]."\n            eval "${commandsArray[$counter]}" &amp;\n            pid=$!\n            pidsArray+=($pid)\n            commandsArrayPid[$pid]="${commandsArray[$counter]}"\n            counter=$((counter+1))\n        done\n\n\n        newPidsArray=()\n        for pid in "${pidsArray[@]}"; do\n            # Handle uninterruptible sleep state or zombies by ommiting them from running process array (How to kill that is already dead ? :)\n            if kill -0 $pid &gt; /dev/null 2&gt;&amp;1; then\n                pidState=$(ps -p$pid -o state= 2 &gt; /dev/null)\n                if [ "$pidState" != "D" ] &amp;&amp; [ "$pidState" != "Z" ]; then\n                    newPidsArray+=($pid)\n                fi\n            else\n                # pid is dead, get it\'s exit code from wait command\n                wait $pid\n                retval=$?\n                if [ $retval -ne 0 ]; then\n                    log "Command [${commandsArrayPid[$pid]}] failed with exit code [$retval]."\n                    retvalAll=$((retvalAll+1))\n                fi\n            fi\n        done\n        pidsArray=("${newPidsArray[@]}")\n\n        # Add a trivial sleep time so bash won\'t eat all CPU\n        sleep .05\n    done\n\n    return $retvalAll\n}\n</code></pre>\n\n<p>Usage:</p>\n\n<pre><code>cmds="du -csh /var;du -csh /tmp;sleep 3;du -csh /root;sleep 10; du -csh /home"\n\n# Execute 2 processes at a time\nParallelExec 2 "$cmds"\n\n# Execute 4 processes at a time\nParallelExec 4 "$cmds"\n</code></pre>\n'}, {'answer_id': 54472111, 'author': 'Skrat', 'author_id': 316622, 'author_profile': 'https://Stackoverflow.com/users/316622', 'pm_score': 2, 'selected': False, 'text': '<p><em>Really</em> late to the party here, but here\'s another solution.</p>\n\n<p>A lot of solutions don\'t handle spaces/special characters in the commands, don\'t keep N jobs running at all times, eat cpu in busy loops, or rely on external dependencies (e.g. GNU <code>parallel</code>).</p>\n\n<p>With <a href="https://stackoverflow.com/a/39189370/316622">inspiration for dead/zombie process handling</a>, here\'s a pure bash solution:</p>\n\n<pre class="lang-sh prettyprint-override"><code>function run_parallel_jobs {\n    local concurrent_max=$1\n    local callback=$2\n    local cmds=("${@:3}")\n    local jobs=( )\n\n    while [[ "${#cmds[@]}" -gt 0 ]] || [[ "${#jobs[@]}" -gt 0 ]]; do\n        while [[ "${#jobs[@]}" -lt $concurrent_max ]] &amp;&amp; [[ "${#cmds[@]}" -gt 0 ]]; do\n            local cmd="${cmds[0]}"\n            cmds=("${cmds[@]:1}")\n\n            bash -c "$cmd" &amp;\n            jobs+=($!)\n        done\n\n        local job="${jobs[0]}"\n        jobs=("${jobs[@]:1}")\n\n        local state="$(ps -p $job -o state= 2&gt;/dev/null)"\n\n        if [[ "$state" == "D" ]] || [[ "$state" == "Z" ]]; then\n            $callback $job\n        else\n            wait $job\n            $callback $job $?\n        fi\n    done\n}\n</code></pre>\n\n<p>And sample usage:</p>\n\n<pre class="lang-sh prettyprint-override"><code>function job_done {\n    if [[ $# -lt 2 ]]; then\n        echo "PID $1 died unexpectedly"\n    else\n        echo "PID $1 exited $2"\n    fi\n}\n\ncmds=( \\\n    "echo 1; sleep 1; exit 1" \\\n    "echo 2; sleep 2; exit 2" \\\n    "echo 3; sleep 3; exit 3" \\\n    "echo 4; sleep 4; exit 4" \\\n    "echo 5; sleep 5; exit 5" \\\n)\n\n# cpus="$(getconf _NPROCESSORS_ONLN)"\ncpus=3\nrun_parallel_jobs $cpus "job_done" "${cmds[@]}"\n</code></pre>\n\n<p>The output:</p>\n\n<pre><code>1\n2\n3\nPID 56712 exited 1\n4\nPID 56713 exited 2\n5\nPID 56714 exited 3\nPID 56720 exited 4\nPID 56724 exited 5\n</code></pre>\n\n<p>For per-process output handling <code>$$</code> could be used to log to a file, for example:</p>\n\n<pre class="lang-sh prettyprint-override"><code>function job_done {\n    cat "$1.log"\n}\n\ncmds=( \\\n    "echo 1 \\$\\$ &gt;\\$\\$.log" \\\n    "echo 2 \\$\\$ &gt;\\$\\$.log" \\\n)\n\nrun_parallel_jobs 2 "job_done" "${cmds[@]}"\n</code></pre>\n\n<p>Output:</p>\n\n<pre><code>1 56871\n2 56872\n</code></pre>\n'}]
['https://Stackoverflow.com/questions/38160', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/163/']