title: Threading optimization. Running many commands on a list of files
body: <p>I have a script that has to run on many files. In each file there are the execution of external commands that generate different files or I need to operate in the same file however in different folders.</p>
  
  <p>I would have something like this:</p>
  
  <pre class="lang-none prettyprint-override"><code>Proc1_File1 -&gt; Proc2.1_File1 -&gt; Proc2.2_File1 -&gt; Proc3.File1 -&gt; Proc4.File1
  Proc1_File2 -&gt; Proc2.1_File2 -&gt; Proc2.2_File2 -&gt; Proc3.File2 -&gt; Proc4.File2
  Proc1_FileN -&gt; Proc2.1_File2 -&gt; Proc2.2_FileN -&gt; Proc3.FileN -&gt; Proc4.FileN
  </code></pre>
  
  <p>So I would have in this example a total of <strong>5</strong> processes, however for example <code>Proc2.1</code> and <code>Proc2.2</code> can run at the same time.</p>
  
  <p>Also I would need to restrict the number of processes that run at the same time, mainly for the <code>Proc2.2</code> and <code>Proc3.3</code> because the external commands are very memory intensive, so running more than 4-5 processes at same time may result in very low memory.</p>
  
  <p>Sorry for this question being so generic, but I don't know where to start and how to solve this problem. As if I have to run every process sequentially it would be too much time consuming.</p>
  
  <p>So, I think that I may need to use a queue. I have been limiting the execution of the external commands using a list and wait like for example here:</p>
  
  <pre><code>__list_commands.append(Popen(__command, shell=True, cwd=str(__p.parent)))
  exit_codes = [p.wait() for p in __list_commands]
  </code></pre>
  
  <p>It would be part of a function that runs the first process. However this makes to wait that a command end to run the next, so it makes sequentially, and I would like to run each process in parallel, and run sequentially each process, first proc1, then proc2...</p>
  
  <p>Thank you and sorry for this so generic question.</p>