title: HttpClient Failing in asp.net web api
body: <p>I have a very strange scenario I keep experiencing on one of my systems.</p>
  
  <p>I have an <code>asp.net web API</code> hosted in IIS. Web requests come into it from mobile devices, and then it posts HTTP requests to another server that executes the specific job and then returns the result. (see flow below) </p>
  
  <p>Phone =>  API => HTTP  => ( Service Executor ) => HTTP  => API  => PHONE  </p>
  
  <p>This works perfectly 95% of the time. However it sometimes gets into this state where NONE of the http requests complete.  </p>
  
  <p>I have observed the following:</p>
  
  <ul>
  <li>After the issues happens once, it will continue to fail indefinitely until i restart the application pool.</li>
  <li>Setting the IIS application pool recycle time to 1 hour has significantly reduced the occurrence. However not totally removed it.</li>
  </ul>
  
  <p>The following method is called from my Asp.net Controller:</p>
  
  <p>Here is my code:</p>
  
  <pre><code>public static async Task&lt;string&gt; MakeRequest(string deviceId, BaseCommand baseCommand)
  {
      using (var client = new HttpClient())
      {
          // Setup Client                
          client.BaseAddress = await GetBaseAddressForDevice(deviceId);
          client.DefaultRequestHeaders.Accept.Clear();
          //Only wait 90 seconds for responses
          client.Timeout = new TimeSpan(0, 0, 0, 90);
  
          var requestId = Guid.NewGuid();
  
          Debug.WriteLine("&gt;Posting  RequestId: " + requestId + " commandId:" + ((int)(baseCommand.CommandId)).ToString(CultureInfo.InvariantCulture) + " data: " + JsonConvert.SerializeObject(baseCommand));
          //
  
          // Make request
          HttpResponseMessage response = null;
          string responseContent = null;
          try
          {
              response = await client.PostAsJsonAsync(((int)baseCommand.CommandId).ToString(CultureInfo.InvariantCulture), baseCommand);
              // read response content
              responseContent = await response.Content.ReadAsStringAsync();
          }
          catch (Exception ex)
          {
              //TODO: Log excepton
              throw ex;
          }
  
          Debug.WriteLine("&gt;Response  RequestId: " + requestId + " data: " + JsonConvert.SerializeObject(baseCommand));
          //
  
          if (response.IsSuccessStatusCode)
          {
              return responseContent;
          }
          HandleErrorResponse(response, responseContent);
          //Wont hit here as an exception will be thrown
          return null;
      }
  }
  </code></pre>
  
  <p>The above example is designed to create and dispose of an <code>HttpClient</code> instance each time. I believe that <code>HttpClient</code> is thread safe and I have tested before with a single instance of <code>HttpClient</code>, and I had the same issues.</p>
  
  <p>Further Behavior that I have observed.</p>
  
  <p>I added logging on the server that <code>Receives</code> the http posts.  When this phenomenon happens, it does receive the HTTP Post, however it then immediately fails. Almost like it establishes the connection, and then fails.</p>
  
  <p>I have some exceptions which Application Insights has collected:</p>
  
  <pre><code> - System.Net.Sockets.SocketException
  An error occurred while sending the request.The underlying connection was closed: An unexpected error occurred on a receive.Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.An existing connection was forcibly closed by the remote host
  </code></pre>
  
  <p>I would really love some input on the issue, maybe some further suggestions to try and debug / location this issue further.</p>