I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives bad robots or robots who do not obey a robots.txt a way to access pages on your site that you do not want accessed. While I am not looking to get into a debate about that I do have a question: If I have a structure like this: /Folder/    /Sub-Folder 1/    /Sub-Folder 2/    (Note: There are no pages within /Folder/ only other folders.)If I Disallow: /Folder/, it will prevent good robots from accessing the directory and any contents within the sub-folders.While we know that bad robots will see the /Folder/ will they be able to see and access the sub-folders and the pages within the subfolders if they are not listed in the robots.txt? (Note: I do not fully understand how good or bad robots crawl a site beyond using a robots.txt and the links within the site.)
Robots.txt and Bad Robots
robots.txt;web crawlers
It depends. Pull up that URL in your browser. Do you see those folders listed? If so, they can find them.To prevent these folders from being listed just add an htaccess file in that directory with this line in it:Options -IndexesThis will prevent the listing of that directory's contents from appearing.