title: How to loop through links in a web page to find all web pages that are a part of the website?
body: <p>I am trying to find all web pages that are a part of a website. For example with the input "www.microsoft.com", I want it to return a list that contains:</p>
  
  <ul>
  <li><p><a href="https://www.microsoft.com" rel="nofollow noreferrer">https://www.microsoft.com</a></p></li>
  <li><p><a href="https://support.microsoft.com" rel="nofollow noreferrer">https://support.microsoft.com</a></p></li>
  <li><p><a href="https://www.microsoft.com/windows" rel="nofollow noreferrer">https://www.microsoft.com/windows</a></p></li>
  </ul>
  
  <p>But not anything out of a website that is NOT a part of the website given, like <a href="https://www.python.org" rel="nofollow noreferrer">https://www.python.org</a></p>
  
  
  
  <p>This is what I am using so far to find ANY webpages, that are connected to a url, using HTML-Agility-Pack:</p>
  
  <pre><code>class CheckWebPages
  {
      public string url;
      private HtmlAgilityPack.HtmlDocument htmlCode;
      public List&lt;string&gt; webpages;
  
      public void Start(string address)
      {
          url = address;
          Navigate(url);
      }
      //Navigate to url
      private void Navigate(String address)
      {
          //Format url
          if (String.IsNullOrEmpty(address)) return;
          if (address.Equals("about:blank")) return;
          if (!address.StartsWith("http://") &amp;&amp; !address.StartsWith("https://"))
          {
              address = "http://" + address;
          }
          try
          {
              url = address;
          }
          catch (System.UriFormatException)
          {
              return;
          }
          //Load HTML from formatted URL
          htmlCode = new HtmlAgilityPack.HtmlWeb().Load(@url);
          //Start Checking Pages
          CheckPages();
      }
  
      private void CheckPages()
      {
          foreach (var a in htmlCode.DocumentNode.Descendants("a"))
          {
              if (a.Attributes["href"] != null)
              {
                  a.Attributes["href"].Value = new Uri(new Uri(url), a.Attributes["href"].Value).AbsoluteUri;
                  webpages.Add(a.Attributes["href"].Value);
              }
          }
      }
  }
  </code></pre>
  
  <p>What I can do but don't know much about:</p>
  
  <ol>
  <li><p>Turn a link like <a href="https://www.microsoft.com" rel="nofollow noreferrer">https://www.microsoft.com</a> into a string like ".microsoft.com"</p></li>
  <li><p>Check if href value contains ".microsoft.com" which works both for pages like something.microsoft.com and www.microsoft.com/something and add the results to list, instead of adding all href.</p></li>
  <li><p>Loop through HTML nodes of all new web pages inside the list, just like going through all the web pages of homepage (or any page) given.</p></li>
  <li><p>Add any unique web page to list, and return all web pages that are found.</p></li>
  </ol>
  
  
  
  
  
  <p>Found a way to get a substring of the given url and then I check if ["href"].Value contains it by using string.Contains()</p>
  
  <pre><code>private string findSubstring(string url)
  {
          string substr = "";
          bool add = false;
          foreach(var i in url.ToCharArray())
          {
              if(i == '/')
              {
                  add = false;
              }
              if(add)
              {
                  substr = substr + i;
              }
              if (i == '.')
              {
                  add = true;
              }
          }
  
          return substr;
  }
  </code></pre>
  
  <p>Since webpages that will be used here will be formatted as:</p>
  
  <p><a href="http://something.website.something/something" rel="nofollow noreferrer">http://something.website.something/something</a>, we just need the ".website.something" part, so it checks until first dot, then until first slash (if there is any) and gets a substring.</p>
  
  <p>I understand that some websites (like this one) doesn't start with www. or something.stackoverflow, so I may find a fix to such examples, but right now it is working as intended.</p>
  
  <p>Now I need to loop through all links there is with this method, and add each to list if the found link doesn't already exist in the list.</p>