title: Screen Scraping, Web Scraping, Web Harvesting, Web Data Extraction, etc. using C# and the .NET Framework
body: <p>I am working on a <code>Microsoft .NET Application in C#</code> for Web Harvesting, Web Scraping, Web Data Extraction, Screen Scraping, etc. Whatever you want to call it. For parsing <code>HTML</code>, I'm attempting to incorporate HTML Agility Pack but it's not as easy as I thought it would be. I have included some specifications and images of what I have so far and was hoping to get your opinions on how I could proceed. basically, I want to do something similar to the layout used in Visual Web Ripper but I have no idea how they do it... Any ideas?</p>
  <p><strong>Specifications:</strong></p>
  <p>My goal is to make a very user friendly point-and-click application for downloading data and images from the web. I would like to load HTML pages using the web browser, and output the parsed data and image links into the text box. The user can specify which HTML tags they want and then download the data into the grid. Finally, export the data into whatever format they need.</p>
  <p>I'm trying to use HTML Agility Pack to load the HTML on the webpage and display it in the textbox.</p>
  <pre class="lang-cs prettyprint-override"><code>// Load Web Browser
  private void Form6_Load(object sender, EventArgs e)
  {
      // Navigate to webpage
      webBrowser.Navigate(&quot;http://www.webopedia.com/TERM/H/HTML.html&quot;);
      // Save URL to memory
      SiteMemoryArray[count] = urlTextBox.Text; 
      // Load HTML from webBrowser
      HtmlWindow window = webBrowser.Document.Window; 
      string str = window.Document.Body.OuterHtml;
      // Extract tags using HtmlAgilityPack and display in textbox
      HtmlAgilityPack.HtmlDocument HtmlDoc = new HtmlAgilityPack.HtmlDocument();
      HtmlDoc.LoadHtml(str);
      HtmlAgilityPack.HtmlNodeCollection Nodes =
          HtmlDoc.DocumentNode.SelectNodes(&quot;//a&quot;);
      foreach (HtmlAgilityPack.HtmlNode Node in Nodes)
      {
          textBox2.Text += Node.OuterHtml + &quot;\r\n&quot;;
      }
  }
  </code></pre>
  <p>Using:</p>
  <pre class="lang-cs prettyprint-override"><code>HtmlWindow window = webBrowser.Document.Window;
  </code></pre>
  <p>I get the error: <code>Object reference not set to an instance of an object</code>.</p>