title: How to get all webpages on a domain
body: <p>I am making a simple <em>web spider</em> and I was wondering if there is a way that can be triggered in my PHP code that I can get all the webpages on a domain...</p>
  
  <p>e.g Lets say I wanted to get all the webpages on Stackoverflow.com . That means that it would get:
  <a href="https://stackoverflow.com/questions/ask">https://stackoverflow.com/questions/ask</a>
  <a href="https://stackoverflow.com/questions/1903214/">pulling webpages from an adult site -- how to get past the site agreement?</a>
  <a href="https://stackoverflow.com/questions/1234214/">https://stackoverflow.com/questions/1234214/</a>
  <a href="https://stackoverflow.com/questions/395261/best-rails-html-parser?rq=1">Best Rails HTML Parser</a></p>
  
  <p>And all the links. How can I get that. Or is there an <strong>API</strong> or <strong>DIRECTORY</strong> that can enable me to get that?</p>
  
  <p>Also is there a way I can get all the <strong>subdomains</strong>?</p>
  
  <p>Btw how do crawlers crawl websites that don't have <a href="http://en.wikipedia.org/wiki/Sitemaps" rel="nofollow noreferrer">SiteMaps</a> or <a href="http://en.wikipedia.org/wiki/Syndication_feed" rel="nofollow noreferrer">Syndication feeds</a>?</p>
  
  <p>Cheers.</p>