title: How to protect/monitor your site from crawling by malicious user
body: <p>Situation:</p>
  
  <ul>
  <li>Site with content protected by username/password (not all controlled since they can be trial/test users)</li>
  <li>a normal search engine can't get at it because of username/password restrictions</li>
  <li>a malicious user can still login and pass the session cookie to a "wget -r" or something else.</li>
  </ul>
  
  <p>The question would be what is the best solution to monitor such activity and respond to it (considering the site policy is no-crawling/scraping allowed)</p>
  
  <p>I can think of some options:</p>
  
  <ol>
  <li>Set up some traffic monitoring solution to limit the number of requests for a given user/IP.</li>
  <li>Related to the first point: Automatically block some user-agents</li>
  <li>(Evil :)) Set up a hidden link that when accessed logs out the user and disables his account. (Presumably this would not be accessed by a normal user since he wouldn't see it to click it, but a bot will crawl all links.)</li>
  </ol>
  
  <p>For point 1. do you know of a good already-implemented solution? Any experiences with it? One problem would be that some false positives might show up for very active but human 
  users.</p>
  
  <p>For point 3: do you think this is really evil? Or do you see any possible problems with it?</p>
  
  <p>Also accepting other suggestions.</p>