Alexa's ia_archiver bot is the main contributor to the Internet Archive's Wayback Machine web collection, and there are advantages to having my website included in that collection. There are other robots that do other useful things too.What's a quick and easy way to make ia_archiver crawl my website more slowly, in order to put less load on the server? I haven't tested the Crawl-delay directive: if you have, and it works, please tell me so. If it doesn't work, please leave a comment. If you've never tested Crawl-delay, then instead please recommend some other solution that takes fifteen minutes or less to implement. Maybe there exists some easy-to-implement software-based solution which will let me throttle too-rapid hits from ia_archiver?Please assume my website is running on Apache 2.4.3 on Debian Linux 6.0.6 on a dedicated server which I administer.
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
web crawlers;robots.txt;cache;internet archive