```

User-Agent: *
Allow: /
Disallow: /posts
Disallow: /posts/
Disallow: /tags
Disallow: /tags/
[...]

```

[Wikipedia says](http://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive):

> 
> Some major crawlers support an Allow
>  directive which can counteract a
>  following Disallow directive. This is
>  useful when you disallow an entire
>  directory but still want some HTML
>  documents in that directory crawled
>  and indexed. While by standard
>  implementation **the first matching
>  robots.txt pattern always wins**,
>  Google's implementation differs in
>  that it first evaluates all Allow
>  patterns and only then all Disallow
>  patterns. (irrelevant in this case
>  since Allow is the first, so google's
>  special behavior does not apply here)
> 
> 
> 

So the `Allow: /` directive overrides **all** the following disallows rendering `robots.txt` effectively useless.

This is the reason all major search engines (bing, google, etc) crawl pages they are not supposed to, like post revisions.

---

> 
> **See also**  
> 
> [Please include question title in title of revision history pages](https://meta.stackexchange.com/questions/44041/please-include-question-title-in-title-of-revision-history-pages/44044#44044)
> 
> 
>