title: Parallel batch file download from Amazon S3 using AWS S3 SDK for .NET
body: <p><strong>Problem:</strong> I would like to download 100 files in parallel from AWS S3 using their .NET SDK. The downloaded content should be stored in 100 memory streams (the files are small enough, and I can take it from there). I am geting confused between Task, IAsyncResult, Parallel.*, and other different approaches in .NET 4.0. </p>
  
  <p><strong>If I try to solve the problem myself</strong>, off the top of my head I imagine something like this pseudocode:
  (edited to add types to some variables)</p>
  
  <pre><code>using Amazon;
  using Amazon.S3;
  using Amazon.S3.Model;
  
  AmazonS3 _s3 = ...;
  IEnumerable&lt;GetObjectRequest&gt; requestObjects = ...;
  
  
  // Prepare to launch requests
  var asyncRequests = from rq in requestObjects 
      select _s3.BeginGetObject(rq,null,null);
  
  // Launch requests
  var asyncRequestsLaunched = asyncRequests.ToList();
  
  // Prepare to finish requests
  var responses = from rq in asyncRequestsLaunched 
      select _s3.EndGetRequest(rq);
  
  // Finish requests
  var actualResponses = responses.ToList();
  
  // Fetch data
  var data = actualResponses.Select(rp =&gt; {
      var ms = new MemoryStream(); 
      rp.ResponseStream.CopyTo(ms); 
      return ms;
  });
  </code></pre>
  
  <p>This code launches 100 requests in parallel, which is good. However, there are two problems:</p>
  
  <ol>
  <li>The last statement will download files serially, not in parallel. There doesn't seem to be BeginCopyTo()/EndCopyTo() method on stream...</li>
  <li>The preceding statement will not let go until all requests have responded. In other words none of the files will start downloading until all of them start.</li>
  </ol>
  
  <p>So here I start thinking I am heading down the wrong path...</p>
  
  <p>Help?</p>