title: Simple screen scraping and analyze in .NET
body: <p>I'm building a small specialized search engine for prise info. The engine will only collect specific segments of data on each site. My plan is to split the process into two steps.</p>
  
  <ol>
  <li><p>Simple screen scraping based on a URL that points to the page where the segment I need exists. <em>Is the easiest way to do this just to use a WebClient object and get the full HTML?</em></p></li>
  <li><p>Once the HTML is pulled and saved analyse it via some script and pull out just the segment and values I need (for example the price value of a product). My problem is that this script somehow has to be unique for each site I pull, it has to be able to handle really ugly HTML (so I don't think XSLT will do ...) and I need to be able to change it on the fly as the target sites updates and changes. I will finally take the specific values and write these to a database to make them searchable</p></li>
  </ol>
  
  <p>Could you please give me some hints on how to architect the best way? Would you do different then described above?</p>