title: Web scraping using WebBrowser in a class library
body: <p>I need to create a method in a class library to get the content of a URL (which may be dynamically populated by JavaScript).</p>
  
  <p>I am clueless, but having googling for the whole day this is what I came up with: (Most of the code is from <a href="https://stackoverflow.com/questions/20930414/how-to-dynamically-generate-html-code-using-nets-webbrowser-or-mshtml-htmldocu/20934538#20934538">here</a>)</p>
  
  <pre><code>using System;
  using System.Threading.Tasks;
  using System.Threading;
  using System.Windows.Forms;
  
  public static class WebScraper
  {
      [STAThread]
      public async static Task&lt;string&gt; LoadDynamicPage(string url, CancellationToken token)
      {
          using (WebBrowser webBrowser = new WebBrowser())
          {
              // Navigate and await DocumentCompleted
              var tcs = new TaskCompletionSource&lt;bool&gt;();
              WebBrowserDocumentCompletedEventHandler onDocumentComplete = (s, arg) =&gt; tcs.TrySetResult(true);
  
              using (token.Register(() =&gt; tcs.TrySetCanceled(), useSynchronizationContext: true))
              {
                  webBrowser.DocumentCompleted += onDocumentComplete;
                  try
                  {
                      webBrowser.Navigate(url);
                      await tcs.Task; // wait for DocumentCompleted
                  }
                  finally
                  {
                      webBrowser.DocumentCompleted -= onDocumentComplete;
                  }
              }
  
              // get the root element
              var documentElement = webBrowser.Document.GetElementsByTagName("html")[0];
  
              // poll the current HTML for changes asynchronosly
              var html = documentElement.OuterHtml;
              while (true)
              {
                  // wait asynchronously, this will throw if cancellation requested
                  await Task.Delay(500, token);
  
                  // continue polling if the WebBrowser is still busy
                  if (webBrowser.IsBusy)
                      continue;
  
                  var htmlNow = documentElement.OuterHtml;
                  if (html == htmlNow)
                      break; // no changes detected, end the poll loop
  
                  html = htmlNow;
              }
  
              // consider the page fully rendered 
              token.ThrowIfCancellationRequested();
              return html;
          }
      }
  }
  </code></pre>
  
  <p>It currently throws this error</p>
  
  <blockquote>
    <p>ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be
    instantiated because the current thread is not in a single-threaded
    apartment.</p>
  </blockquote>
  
  <p>Am I close? Is there a fix for the above?</p>
  
  <p>Or if I am off the track, is there a ready solution to get dynamic web content using .NET (that can be called from a class library)?</p>