title: How many processes should I run in parallel?
body: <p>I have a parallelized task that reads stuff from multiple files, and writes it out the information to several files.</p>
  
  <p>The idiom I am currently using to parallelize stuff:</p>
  
  <pre><code>listOfProcesses = []
  for fileToBeRead in listOfFilesToBeRead:
      process = multiprocessing.Process(target = somethingThatReadsFromAFileAndWritesSomeStuffOut, args = (fileToBeRead))
      process.start()
      listOfProcesses.append(process)
  
  for process in listOfProcesses:
      process.join()
  </code></pre>
  
  <p>It is worth noting that <code>somethingThatReadsFromAFileAndWritesSomeStuffOut</code> might itself parallelize tasks (it may have to read from other files, etc. etc.).</p>
  
  <p>Now, as you can see, the number of processes being created doesn't depend upon the number of cores I have on my computer, or anything else, <em>except for how many tasks need to be completed</em>. If ten tasks need to be run, create ten processes, and so on.</p>
  
  <p>Is this the best way to create tasks? Should I instead think about how many cores my processor has, etc.?</p>