title: Finding sub-directories on a web server
body: <p>We can easily find subdirectories on our local disc using os.walk() but what if those directories are not local and are on a web server? </p>
  
  <p>For example, I have a website called <a href="http://www.geoglobaldomination.org" rel="nofollow">http://www.geoglobaldomination.org</a>. There are a couple subdirectories that are NOT referenced on the home page. eg.<a href="http://www.geoglobaldomination.org/kml" rel="nofollow">http://www.geoglobaldomination.org/kml</a> and <a href="http://www.geoglobaldomination.org/kml/temp" rel="nofollow">http://www.geoglobaldomination.org/kml/temp</a>.</p>
  
  <p>How can I find those subdirectories using a simple python crawler without using the HTML tags as a reference point?</p>