title: How to read sitemap url text from robots.txt file
body: <p>I want to read the text of robots.txt file(www.abcd.com/robots.txt) which contains the sitemap url from my C# application. I have to use if else to generate Alerts if sitemap url present in robots.txt file then it displays yes and it does not contain sitemap url then it will display no.</p>
  
  <p>Robots.txt file text look like this:</p>
  
  <pre><code># Crawlers Setup
  User-agent: *
  Disallow:
  Crawl-delay: 10
  
  # Website Sitemap
  Sitemap: http://www.abcd.com/sitemap.xml
  </code></pre>
  
  <p>How can I read this sitemap text from robots.txt file as robots.txt is also a link not actually a text file. It is www.abcd.com/robots.txt</p>