title: Web Crawler C# .Net
body: <p>I'm not sure if this is actually called the web crawler, but this is what I'm trying to do.</p>
  
  <p>I'm building a program in visual studio 2010 using C# .Net.</p>
  
  <p>I want to find all the urls that has the same first part.</p>
  
  <p>Say I have a homepage: <code>www.mywebsite.com</code>, and there are several subpage: <code>/tab1</code>, <code>/tab2</code>, <code>/tab3</code>, etc.</p>
  
  <p>Is there a way to get a list of all urls that begins with <code>www.mywebsite.com</code>?</p>
  
  <p>So by providing <code>www.mywebsite.com</code>, the program returns <code>www.mywebsite.com/tab1</code>, <code>www.mywebsite.com/tab2</code>, <code>www.mywebsite.com/tab3</code>, etc.</p>
  
  <p>ps. I do not know how many total sub pages there are.</p>
  
  <p>--edit at 12:04pm--</p>
  
  <p>sorry for the lack of explanation.</p>
  
  <p>I want to know how to write a crawler in C# that do the above task.</p>
  
  <p>All I know is the main url <code>www.mywebsite.com</code>, and the goal is to find all its sub pages.</p>
  
  <p>-- edit at 12:16pm--</p>
  
  <p>Also, there is no links on the main page, the html is basically blank.</p>
  
  <p>I just know that the subpages exist, but have no way to link to it except for providing the exact urls.</p>