title: ASP.Net problem validating 1000's external URLs
body: <p>I have a database of 1000's of external urls that I want to check every month or so. Most the urls are different hosts/domains. The problem is it seems to slow down after 100's of URLs.</p>
  <p>Today I'm using a singleton HTTP client  in .Net Standard 2.0 library with a backport of SocketsHttpHandler. I have the same issue locally and in my Azure hosted version.</p>
  <p>I setup sockets as follows :</p>
  <pre><code>.AllowAutoRedirect = False
  .UseProxy = False
  .Proxy = Nothing
  .UseCookies = False
  .MaxConnectionsPerServer = 50
  .SslOptions = sslOptions
  .ConnectTimeout = New TimeSpan(0, 0, 5) 
  .PooledConnectionIdleTimeout = New TimeSpan(0, 0, 2)
  </code></pre>
  <p>I initiate my client as follows :</p>
  <pre><code>_Client = New HttpClient(sockets, False)
  </code></pre>
  <p>I use HttpRequestMessage to make the request and HttpResponseMessage to handle it. These both get disposed in a finally block.</p>
  <p>I'm using a GET request. I did Experiment with HEAD but I found not all servers responded properly.</p>
  <p>Adding the below to try and ensure connections get closed :</p>
  <pre><code>req.Headers.TryAddWithoutValidation(&quot;Connection&quot;, &quot;close&quot;)
  </code></pre>
  <p>I also use ResponseHeadersRead to improve performance and remove the need to read the response stream before the headers which then enables me to close the connection.</p>
  <pre><code>response = Await _Client.SendAsync(req, HttpCompletionOption.ResponseHeadersRead)
  </code></pre>
  <p>I'm wondering if it is something to do with how TCP connections work under the hood, but having read various posts I have run out of optimizations and ideas.</p>