From what I learned, there's one way I could cause all URLs on a domain strictly serving images not to be indexed and that is with the x-robots-tag HTTP header. Now I check my logs and find out that google and even baidu are downloading the entire contents of the image URLs. I was hoping they'd stop downloading when they came across this line:X-Robots-Tag: noindex, noimageindexEither I formatted that line wrong (used wrong casing or wrong order of values or something), or search engines are just plain dumb and they just decide to download everything just to waste customer's money.I looked into robots.txt and thought of using the noindex line but when I did, google complained about having no access to what they call an important url when it isn't important.I don't want to block their IP's because I have text-based content on another domain running on the same server that I want them to index.I'm tempted to offer search engines the equivalent of what users get if they requested the URL via the HEAD method (full headers but no actual content), but I might get penalized for content cloaking. Is there something I can do to rectify this?
How to tell search engines to not index entire image domain without making them waste server bandwidth or making google complain
google;images;indexing;download;baidu