title: Robots.txt - What is the proper format for a Crawl Delay for multiple user agents?
body: <p>Below is a sample <strong>robots.txt</strong> file to Allow <strong>multiple user agents</strong> with <strong>multiple crawl delays</strong> for each user agent. The Crawl-delay values are for illustration purposes and will be different in a real robots.txt file.</p>
  
  <p>I have searched all over the web for proper answers but could not find one. There are too many mixed suggestions and I do not know which is the correct / proper method.</p>
  
  <p><strong>Questions:</strong></p>
  
  <p>(1) Can each user agent have it's own crawl-delay? (I assume yes)</p>
  
  <p>(2) Where do you put the crawl-delay line for each user agent, before or after the Allow / Dissallow line?</p>
  
  <p>(3) Does there have to be a blank like between each user agent group.</p>
  
  <p>References:</p>
  
  <p><a href="http://www.seopt.com/2013/01/robots-text-file/" rel="noreferrer">http://www.seopt.com/2013/01/robots-text-file/</a></p>
  
  <p><a href="http://help.yandex.com/webmaster/?id=1113851#1113858" rel="noreferrer">http://help.yandex.com/webmaster/?id=1113851#1113858</a></p>
  
  <p>Essentially, I am looking to find out how the final robots.txt file should look using the values in the sample below. </p>
  
  <p>Thanks in advance.</p>
  
  <pre><code># Allow only major search spiders    
  User-agent: Mediapartners-Google
  Disallow:
  Crawl-delay: 11
  
  User-agent: Googlebot
  Disallow:
  Crawl-delay: 12
  
  User-agent: Adsbot-Google
  Disallow:
  Crawl-delay: 13
  
  User-agent: Googlebot-Image
  Disallow:
  Crawl-delay: 14
  
  User-agent: Googlebot-Mobile
  Disallow:
  Crawl-delay: 15
  
  User-agent: MSNBot
  Disallow:
  Crawl-delay: 16
  
  User-agent: bingbot
  Disallow:
  Crawl-delay: 17
  
  User-agent: Slurp
  Disallow:
  Crawl-delay: 18
  
  User-agent: Yahoo! Slurp
  Disallow:
  Crawl-delay: 19
  
  # Block all other spiders
  User-agent: *
  Disallow: /
  
  # Block Directories for all spiders
  User-agent: *
  Disallow: /ads/
  Disallow: /cgi-bin/
  Disallow: /scripts/
  </code></pre>
  
  <p>(4) If I want to set all of the user agents to have crawl delay of 10 seconds, would the following be correct?</p>
  
  <pre><code># Allow only major search spiders
  User-agent: *
  Crawl-delay: 10
  
  User-agent: Mediapartners-Google
  Disallow:
  
  User-agent: Googlebot
  Disallow:
  
  User-agent: Adsbot-Google
  Disallow:
  
  User-agent: Googlebot-Image
  Disallow:
  
  User-agent: Googlebot-Mobile
  Disallow:
  
  User-agent: MSNBot
  Disallow:
  
  User-agent: bingbot
  Disallow:
  
  User-agent: Slurp
  Disallow:
  
  User-agent: Yahoo! Slurp
  Disallow:
  
  # Block all other spiders
  User-agent: *
  Disallow: /
  
  # Block Directories for all spiders
  User-agent: *
  Disallow: /ads/
  Disallow: /cgi-bin/
  Disallow: /scripts/
  </code></pre>