title: How do I only allow crawlers to visit a part of the site?
body: <p>I've got an ajax rich website which has extensive _escaped_fragment_ portions for Ajax indexing. While all my _escaped_fragment_ urls do 301 redirects to a special module which then outputs the HTML snapshots the crawlers need (i.e. <code>mysite.com/#!/content</code> redirects to <code>mysite.com/?_escaped_fragment_=/content</code> which in turn 301s to <code>mysite.com/raw/content</code>), I'm somewhat afraid of users stumbling on those "raw" URLs themselves and making them appear in search engines.</p>
  
  <p>In PHP, how do I make sure only robots can access this part of the website? (much like StackOverflow disallows its sitemap to normal users, and only lets robots access it)</p>