title: Robots.txt block all except Google and partners
body: <p>I am trying to block all bots from my site and allow only Google and its partners to have access (except one folder).</p>
  
  <p>This is my robots.txt</p>
  
  <p>However, I see from my log that twitterbot is able to access my site </p>
  
  <pre><code>GET /robots.txt HTTP/1.1" 200 560 "-" "Twitterbot/1.0"
  </code></pre>
  
  <p>Here is my txt</p>
  
  <pre><code>User-agent: Googlebot
  Disallow: /administrator
  Allow: /
  
  
  User-agent: Googlebot-Mobile
  Disallow: /administrator
  Allow: /
  
  User-agent: Mediapartners-Google
  Disallow: /administrator
  Allow: /
  
  User-agent: Googlebot-news
  Disallow: /administrator
  Allow: /
  
  User-agent: Googlebot-Image
  Disallow: /administrator
  Allow: /
  
  User-agent: AdsBot-Google
  Disallow: /administrator
  Allow: /
  
  User-agent: AdsBot-Google-Mobile
  Disallow: /administrator
  Allow: /
  
  User-agent: APIs-Google
  Disallow: /administrator
  Allow: /
  
  User-agent: *
  Disallow: /
  </code></pre>