The robots.txt file, its syntax and its role.
The robots.txt file is intended for the "spiders". Spiders are programs which explore the Web and enable search engines to discover your site and to analyze its contents.
- prohibit the exploration of certain pages to some spiders.
Note that the "robots" meta tag can also be added in each one of your pages to prohibit indexing.
Comment. This will be ignored by robots.
This mention can be followed by * or by the exact name of an existing spider.
This mention can be followed by only ONE repertory or file name.
where AAAAAAA and AAAAAAA' are the names of the robots and BBBBBBB, BBBBBBB ', CCCCCCC and CCCCCCC' the names of the files and/or repertories that you wish to hide to these robots.
If you use the * instead of the name of a robot, the following lines will be regarded as prohibitions of indexing for ALL the robots.
If you use the "/" instead of the file name, NO file of the site will be indexed.
LinkSpirit is a free utility, downloadable on this site, that enables you to create or edit easily "robots" Meta-tags and the "robot.txt" file.
This utility carries out a checking of the syntax of your Robots.txt file by taking into account the rules appearing on: http://www.robotstxt.org/wc/norobots.html and the robots listed on: http://www.robotstxt.org/wc/active/html/index.html.
If you wish to proceed manually, you just need a text editor (Wordpad for example) to create a text file (with the .txt extension).
Here is a typical example of what can be the contents of the file robots.txt.
c) Disallow:/sources/ none of the files contained in the "sources" repertory can be indexed.
d) Disallow:/admin/perso/ none of the files contained in the "admin/perso" repertory can be indexed.
Note: During the transfer of this file to your FTP, be sure to use the ASCII transfert mode.
a) Only one robots.txt file must exist on the whole of your site. It must be located at the root of it.
b) If you wish to impose different rules for each search engine, you can (and you must) create several User-agent sections.
c) The name of the file (robots.txt) must be imperatively be written in low case.
d) Register only one directory or one file name behind the Disallow order. Disallow syntax: "file1.htm, file2.htm" is not authorized neither is "Dir1/, Dir2/".
E) Transfer your robots.txt file in ASCII mode. Many ftp-client modify the code of the txt files when they are not transferred in ASCII mode. This is the cause of the most frequently encountered problems with the robots.txt file.
b) "Allow" field does not exist.
b) The "Allow" field exists and enables to make exceptions to a general prohibition.
CAUTION: "Google rules" can make your robots.txt file incomprehensible to other robots if they appear in a "User-agent: *" zone. Thus allways put those particular instructions after "User-agent: Googlebot".