title: HttpWebRequest Limitations? Or bad implementation
body: <p>I am trying to build a c# console app that will monitor about 3000 urls (Just need to know that HEAD request returned 200, not necessarily content, etc.)</p>
  <p>My attempt here was to build a routine the checks the web URLS, looping and creating threads each executing the routine. What's happening is if i run with &lt;20 threads, it executes ok most of the time, but if i use &gt;20 threads, some of the url's time out. I tried increasing the Timeout to 30 seconds, same occurs. The network I am running this on is more than capable of executing 50 HTTP HEAD requests (10MBIT connection at ISP), and both the CPU and network run very low when executing the routine.</p>
  <p>When a timeout occurs, i test the same IP on a browser and it works fine, I tested this repeatedly and there was never a case during testing that a &quot;timed out&quot; url was actually timing out.</p>
  <p>The reason i want to run &gt;20 threads is that i want to perform this test every 5 minutes, with some of the URL's taking a full 10sec (or higher if the timeout is set higher), i want to make sure that its able to run through all URLs within 2-3 minutes.</p>
  <p>Is there a better way to go about checking if a URL is available, or, should I be looking at the system/network for an issue.</p>
  <h1>MAIN</h1>
  <pre><code>        while (rdr.Read())
          {
              Thread t = new Thread(new ParameterizedThreadStart(check_web));
  
              t.Start(rdr[0]);
              
  
          }
  </code></pre>
  <hr />
  <pre><code>      static void check_web(object weburl)
        {
            bool isok;
            isok = ConnectionAvailable(weburl.ToString());
        }
  
  
  
        public static bool ConnectionAvailable(string strServer)
        {
  
            try
            {
                strServer = &quot;http://&quot; + strServer;
                HttpWebRequest reqFP = (HttpWebRequest)HttpWebRequest.Create(strServer);
                reqFP.Timeout = 10000;
                reqFP.Method = &quot;HEAD&quot;;
  
                HttpWebResponse rspFP = (HttpWebResponse)reqFP.GetResponse();
                if (HttpStatusCode.OK == rspFP.StatusCode)
                {
                    Console.WriteLine(strServer + &quot; - OK&quot;);
                    rspFP.Close();
                    return true;
                }
                else
                {
                    Console.WriteLine(strServer + &quot; Server returned error..&quot;);
                    rspFP.Close();
                    return false;
  
                }
  
            }
  
            catch (WebException x)
            {
                if (x.ToString().Contains(&quot;timed out&quot;))
                {
                    Console.WriteLine(strServer + &quot; - Timed out&quot;);
                }
                else
                {
                    Console.WriteLine(x.Message.ToString());
                }
  
                return false;
  
            }
            
        }
  </code></pre>