title: How to fetch API data using WebClient and multiple threads?
body: <p>So I am trying to query an API that's accessible via HTTP ( no authorization ). To speed things up, I tried to use a Parallel.ForEach loop but it seems like the longer it runs, the more errors pop up.</p>
  
  <p>It fails to retrieve more and more requests. I know the API provider isn't limiting me because I can request the very same blocked URLs in my Internet browser. Also, these are different failed URLs each time, so it doesn't seem to be the case of malformed requests.</p>
  
  <p>The error doesn't seem to occur while I use single threaded foreach loop.</p>
  
  <p>My malfunctioning loop is below:</p>
  
  <pre><code>Parallel.ForEach(this.urlArray, singleUrl =&gt; {
  this.apiResponseBlob = new System.Net.WebClient ().DownloadString(singleUrl );
  this.responsesDictionary.Add(singleUrl, apiResponseBlob);
  }
  </code></pre>
  
  <p>Normal foreach loop works fine but is very slow:</p>
  
  <pre><code>foreach (string singleUrl in this.urlArray) {
  this.apiResponseBlob = new System.Net.WebClient ().DownloadString(singleUrl);
  this.responsesDictionary.Add(singleUrl, apiResponseBlob);
  }
  </code></pre>
  
  <p>Also: I've had a solution in PHP - I spawned several "fetchers" simultaneously and it never hung up. It seems strange to me that PHP would handle multithreaded retrieval better than C# so I must obviously miss something.</p>
  
  <p>How do I query the API fastest way? Without these strange failures?</p>