title: fastest method of large file processing using concurrent futures python 3.5
body: <p>I am trying to grasp multithreading/multiprocessing using concurrent futures. </p>
  
  <p>I have tried using the following sets of code. I understand that I will always have the disk IO problem, but I want to maximize my ram and CPU usage to the fullest extent possible. </p>
  
  <p>What method is the most used/best method for large scale processing? </p>
  
  <p>How do you use concurrent futures for processing large datasets?</p>
  
  <p>Is there a more preferred method than the ones below?</p>
  
  <p>Method 1:</p>
  
  <pre><code>for folders in os.path.isdir(path):
      p = multiprocessing.Process(pool.apply_async(process_largeFiles(folders)))
      jobs.append(p)
      p.start()
  </code></pre>
  
  <p>Method 2:</p>
  
  <pre><code>with concurrent.futures.ThreadPoolExecutor(max_workers=100) as executor:
      for folders in os.path.isdir(path):
          executor.submit(process_largeFiles(folders), 100)
  </code></pre>
  
  <p>Method 3:</p>
  
  <pre><code>with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
      for folders in os.path.isdir(path):
          executor.submit(process_largeFiles(folders), 10)
  </code></pre>
  
  <p>Should I attempt to  use process pool and thread pool together? </p>
  
  <p>Method (thought):</p>
  
  <pre><code>with concurrent.futures.ProcessPoolExecutor(max_workers=10) as process:
       with concurrent.futures.ThreadPoolExecutor(max_workers=100) as thread:
            for folders in os.path.isdir(path):
                process.submit(thread.submit(process_largeFiles(folders), 100),10)
  </code></pre>
  
  <p>What is the most efficient method to maximize my ram and cpu in the broadest use case? </p>
  
  <p>I am aware that starting processes takes a bit of time, but would it be outweighed with the size of my files being processed? </p>