Meta tags and headers In addition to root-level robots.txt files, robots exclusion directives can be applied at a more granular level through the use of Robots meta tags and X-Robots-Tag HTTP headers. The robots meta tag cannot be used for non-HTML files such as images, text files, or PDF documents. On the other hand, the X-Robots-Tag can be added to non-HTML files by using .htaccess and httpd.conf files.
meta tags and headers in addition to root-level robotstxt files robots exclusion directives can be applied at a more granular level through the use of robots meta tags and x-robots-tag http headers the robots meta tag cannot be used for non-html files such as images text files or pdf documents on the other hand the x-robots-tag can be added to non-html files by using htaccess and httpdconf files