title: Parallel approach for large batch processing involving a lot of disk I/O
body: <p>I've got a task where I need to do the following</p>
  
  <pre><code>for fi in files, do in parallel:
    for job in jobs, do serially:
      read fi into memory from disk
      do job(fi), save output to disk
  </code></pre>
  
  <p>(Peculiarities of the task make it impractical to parallelize the inner loop.)</p>
  
  <p>There are thousands of <code>files</code> and about 400 <code>jobs</code>.  Right now the <code>files</code> are a bunch of .csvs, and I'm saving the output to a bunch of csv's elsewhere.  input csvs are about 30MB, outputs are about 10MB</p>
  
  <p>What I really want is a database, but the databases that I know about can't handle parallel I/O.  So I'm just reading from and saving a bunch of csv's.  I know that the OS can't do parallel I/O either, but it handles parallel requests better than the databases that I know about.</p>
  
  <p>So, what's a better way to do this?  I'm on AWS, and right now everything is sitting in an EBS volume.  Would I expect better performance pushing all the inputs up to an S3 bucket, downloading them as needed, and uploading the outputs (the files are a couple of MB each)?</p>
  
  <p>Or is there some sort of parallel I/O database architecture that lives on the cloud somewhere that I can learn about and then rent?</p>
  
  <p>Suggestions for how to approach this are appreciated.</p>