title: await thousands of Tasks
body: <p>I have an application which converts some data often there are 1.000 - 30.000 files.</p>
  
  <p>I need to do 3 steps:</p>
  
  <ol>
  <li>copy a File (replace some text in there)</li>
  <li>Make a Webrequest with WebClient to download a file (I send the copied file to a WebServer, which converts the file to another format)</li>
  <li>Take the downloaded file and change some of the content</li>
  </ol>
  
  <p>So all three steps include some I/O and I used async/await methods:</p>
  
  <pre><code>var tasks = files.Select(async (file) =&gt;
  {
      Item item = await createtempFile(file).ConfigureAwait(false);
      await convert(item).ConfigureAwait(false);
      await clean(item).ConfigureAwait(false);
  }).ToList();
  
  await Task.WhenAll(tasks).ConfigureAwait(false);
  </code></pre>
  
  <p>I don´t know if this is the best practice, because I create more than thousand tasks. I thought about splitting the three steps like:</p>
  
  <pre><code>List&lt;Item&gt; items = new List&lt;Item&gt;();
  var tasks = files.Select(async (file) =&gt;
  {
      Item item = await createtempFile(file, ext).ConfigureAwait(false);
      lock(items)
          items.Add(item);
  }).ToList();
  
  await Task.WhenAll(tasks).ConfigureAwait(false);
  
  var tasks = items.Select(async (item) =&gt;
  {
      await convert(item, baseAddress, ext).ConfigureAwait(false);
  }).ToList();
  
  await Task.WhenAll(tasks).ConfigureAwait(false);
  
  var tasks = items.Select(async (item) =&gt;
  {
      await clean(targetFile, item.Doctype, ext).ConfigureAwait(false);
  }).ToList();
  
  await Task.WhenAll(tasks).ConfigureAwait(false);
  </code></pre>
  
  <p>But that doesn´t seem to be better or faster, because I create 3 times thousands of tasks.</p>
  
  <p>Should I throttle the creation of tasks? Like chunks of 100 tasks?
  Or am I just overthinking it and the creation of thousands of tasks is just fine.</p>
  
  <p>The CPU is idling with 2-4% peak, so I thought about too many awaits or context switches.</p>
  
  <p>Maybe the WebRequest calls are too many, because the WebServer/WebService can´t handle thousands of Requests simultaneously and I should only throttle the WebRequests?</p>
  
  <p>I already increased the .NET maxconnection in the app.config file.</p>