title: How to get improve latency when using HtmlAgilityPack to scan a few web links in real-time?
body: <p>Here is the method I call every second but when I measure the time it takes to complete it, sometimes it takes less than a second but other times up to 4 minutes.</p>
  
  <pre><code>HashSet&lt;int&gt; Handles = new HashSet&lt;int&gt; ( );
  
  void MonitorWebsite ( )
  {
      string linkIdPrefix = "thread_title_";
      foreach ( var lnk in this.Links )
      {
          doc = web.Load ( lnk );
  
          var threadLinks = doc.DocumentNode.Descendants ( "a" )
                                          .Where ( link =&gt; link.Id.StartsWith ( linkIdPrefix ) );
  
          foreach ( var topic in threadLinks )
          {
              string id = topic.Id.Substring ( linkIdPrefix.Length ); // remove "thread_title_"
              string name = topic.InnerHtml; // or link.InnerText
  
              int handle = Convert.ToInt32 ( id );
              if ( this.Handles.Contains ( handle ) == false &amp;&amp; name.IndexOf ( "keyword", StringComparison.OrdinalIgnoreCase ) &gt;= 0 )
              {
                  string href = topic.GetAttributeValue ( "href", null );
  
                  this.OpenInBrowser ( href );
                  SoundPlayer player = new SoundPlayer ( Properties.Resources.Success );
                  player.Play ( );
              }
              this.Handles.Add ( handle );
          }
      }
  }
  </code></pre>
  
  <p>How can I improve it so that it takes less time? Is it out of my control? This is a forum website and if my program finishes late then it opens the link late, like 1-4 minutes after the topic has been opened, which in most cases is too late, as someone else already acted before me.</p>
  
  <p>Also I have 3 links in <code>this.Links</code> and for each of them <code>threadLinks</code> contain about 40 topics on one page which is what I am reading to see if there is a new topic opened. Not sure if going through these 120 makes a difference.</p>
  
  <p>Would there be a way to maybe only get a fraction of the page?</p>