A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
Does Yahoo Slurp crawl hammer a site?
search engines;robots.txt;web crawlers
Hammer is subjective, but there does seem to be a general belief that Slurp is let's say...maybe a little more aggressive than it really needs to be, and that it's been going on for years, ie. 2007, 2009, 2011. A basic search for something like slurp robots delay will turn up a giant pile of complaints.In case you're tempted to actually block Slurp, as suggested at one of those links, you might want to also see this recent question here for a different take.