title: HttpClient GetAsync ThreadPool Starvation
body: <p>We have a microservice oriented backend stack. All of the microservices built top of <code>Nancy</code> and registered as windows services with <code>topshelf</code>.</p>
  
  <p>One of the service, which handles most traffic (~5000 req/s), started to have threadpool starvation problem on 3 out of 8 servers.</p>
  
  <p>This is the exception we are getting when hitting a specific endpoint:</p>
  
  <pre><code>System.InvalidOperationException: There were not enough free threads in the ThreadPool to complete the operation.
     at System.Net.HttpWebRequest.BeginGetResponse(AsyncCallback callback, Object state)
     at System.Net.Http.HttpClientHandler.StartGettingResponse(RequestState state)
     at System.Net.Http.HttpClientHandler.StartRequest(Object obj)
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at RandomNamedClient.&lt;GetProductBySkuAsync&gt;d__20.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at ProductService.&lt;GetBySkuAsync&gt;d__3.MoveNext() in ...\ProductService.cs:line 34
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at ProductModule.&lt;&gt;c__DisplayClass15.&lt;&lt;.ctor&gt;b__b&gt;d__1d.MoveNext() in ...\ProductModule.cs:line 32
  </code></pre>
  
  <p>This endpoint calls another service -which is out of my team's domain- in order to get product data. Implementation of it as follows:</p>
  
  <pre><code>Get["/product/sku/{sku}", true] = async (parameters, ctx) =&gt;
  {
      string sku = parameters.sku;
      var product = await productService.GetBySkuAsync(sku);
      return Response.AsJson(new ProductRepresentation(product));
  };
  </code></pre>
  
  <p><code>ProductService.GetBySkuAsync(string sku)</code> implementation:</p>
  
  <pre><code>public async Task&lt;Product&gt; GetBySkuAsync(string sku)
  {
      var productDto = await randomNamedClient.GetProductBySkuAsync(sku);
      if (productDto == null)
      {
          throw new ProductDtoNotFoundException("sku", sku);
      }
  
      var variantDto = productDto.VariantList.FirstOrDefault(v =&gt; v.Sku == sku);
  
      if (variantDto == null)
      {
          throw new ProductVariantDtoNotFoundException("sku", sku);
      }
  
      return MapVariantDtoToProduct(variantDto, productDto);
  }
  </code></pre>
  
  <p><code>RandomNamedClient.GetProductBySkuAsync(string sku)</code> implementation (it's from an internal package):</p>
  
  <pre><code>public async Task&lt;ProductDto&gt; GetProductBySkuAsync(string sku)
  {
    HttpResponseMessage result = await this._serviceClient.GetAsync("Product?Sku=" + sku);
    return result == null || result.StatusCode != HttpStatusCode.OK ? (ProductDto) null : this.Decompress&lt;ProductDto&gt;(result);
  }
  </code></pre>
  
  <p><code>RandomNamedClient.Decompress&lt;T&gt;(HttpResponseMessage response)</code> implementation:</p>
  
  <pre><code>private T Decompress&lt;T&gt;(HttpResponseMessage response)
  {
    if (!response.Content.Headers.ContentEncoding.Contains("gzip"))
      return HttpContentExtensions.ReadAsAsync&lt;T&gt;(response.Content).Result;
    using (GZipStream gzipStream = new GZipStream((Stream) new MemoryStream(response.Content.ReadAsByteArrayAsync().Result), CompressionMode.Decompress))
    {
      byte[] buffer = new byte[8192];
      using (MemoryStream memoryStream = new MemoryStream())
      {
        int count;
        do
        {
          count = gzipStream.Read(buffer, 0, 8192);
          if (count &gt; 0)
            memoryStream.Write(buffer, 0, count);
        }
        while (count &gt; 0);
        return JsonConvert.DeserializeObject&lt;T&gt;(Encoding.UTF8.GetString(memoryStream.ToArray()));
      }
    }
  }
  </code></pre>
  
  <p>All of our services built as Release/32-bit. We didn't tweak anything about threadpool usage.</p>