title: Apache Nutch does not index the entire website, only subfolders
body: <p>Apache Nutch 1.2 does not index the entire website, only subfolders. My index-page provides links in most areas/subfolders of my website. For example stuff, students, research... But nutch only crawl in one specific folder - "students" in this case. Seems as if links in other directories are not followed.</p>
  
  <p>crawl-urlfilter.txt:
  +^http://www5.my-domain.de/</p>
  
  <p>seed.txt in the URLs-folder:
  <a href="http://www5.my-domain.de/" rel="nofollow">http://www5.my-domain.de/</a></p>
  
  <p>Starting nutch with(windows/linux both used):
  nutch crawl "D:\Programme\nutch-1.2\URLs" -dir "D:\Programme\nutch-1.2\crawl" -depth 10 -topN 1000000</p>
  
  <p>Different variants for depth(5-23) and topN(100-1000000) are tested. Providing more links in seed.txt doesnt help at all, still not following links found in injected pages.</p>
  
  <p>Interestingly, crawling gnu.org works perfect. No robots.txt or preventing meta-tags used in my site.</p>
  
  <p>Any ideas?</p>