title: Running a limited number of child processes in parallel in bash?
body: <p>I have a large set of files for which some heavy processing needs to be done.
  This processing in single threaded, uses a few hundred MiB of RAM (on the machine used to start the job) and takes a few minutes to run.
  My current usecase is to start a hadoop job on the input data, but I've had this same problem in other cases before.</p>
  
  <p>In order to fully utilize the available CPU power I want to be able to run several those tasks in paralell.</p>
  
  <p>However a very simple example shell script like this will trash the system performance due to excessive load and swapping:</p>
  
  <pre><code>find . -type f | while read name ; 
  do 
     some_heavy_processing_command ${name} &amp;
  done
  </code></pre>
  
  <p>So what I want is essentially similar to what "gmake -j4" does.</p>
  
  <p>I know bash supports the "wait" command but that only waits untill all child processes have completed. In the past I've created scripting that does a 'ps' command and then grep the child processes out by name (yes, i know ... ugly).</p>
  
  <p>What is the simplest/cleanest/best solution to do what I want?</p>
  
  <hr>
  
  <p>Edit: Thanks to Frederik: Yes indeed this is a duplicate of <a href="https://stackoverflow.com/questions/6511884/how-to-limit-number-of-threads-used-in-a-function-in-bash">How to limit number of threads/sub-processes used in a function in bash</a>
  The "xargs --max-procs=4" works like a charm.
  (So I voted to close my own question)</p>