I am currently using the Robots.txt file (the standard one of Joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.Is there a reason to use one over the other?
Should I use both Robots.txt and META NAME=ROBOTS?
robots.txt;meta robots
The meta tags are sometimes useful as they offer more fine-grained guidelines for robots than robots.txt. They can also be used if you do not have access to the root directory of the server.However, the robots.txt is more powerful, as it can be used to restrict access to entire directories (while the scope of a meta tag is always constrained to the single file it appears in).If you use both, be aware that a well-behaved robot will not even see a meta tag if robots.txt disallow crawling the directory where it appear.  I.e. there is nothing to be gained in adding meta tags (for finer resolution, etc.)  to files that are in areas of the site disallowed by robots.txt.  If you for some reason want to use meta tags, you must allow crawling of the directory they appear for them to be obeyed.And while there is no harm done using both, there is no need to add meta tags if your robots.txt does what you want.