title: Is it possible to fake Useragent in C#?
body: <p>I want to create a crawler with C#. The problem is that some websites have disabled black listed crawlers in their <em>robots.txt</em> file, using:</p>
  
  <pre><code>User-agent: *
  Disallow: /
  </code></pre>
  
  <p>Is there a way I could fake my request to show that I'm for instance Googlebot?</p>