title: Is there a test case available (simulator) for search engine bots
body: <p>I have wrote myself a very strong protection class "BlockIp" that can use a blacklist with ip's and can detect strange IP configuration's and can block proxies. When it found one, i get a detailed email about the visitor and why it is blocked and what they want to try to do (once a day of course). It seems that it is working very well because i received some real attacks in the past that have been blocked by this class. It does not block legal bots, but that is not easy to test that the detection method is correct. </p>
  
  <p>Today i get an email from the class that it has blocked "ycar10.mobile.bf1.yahoo.com", it identifies itself as a yahoo robot but was behind a proxy. I search the net if it was blacklisted but didn't found that it is blacklisted. So the question is: Is it right to block bots behind a proxy (use legal bots proxy's anyway?)? Here some information about the bot:</p>
  
  <pre><code>HTTP_ACCEPT = */*
  HTTP_X_FORWARDED_FOR = 107.38.3.137,  98.137.88.60
  HTTP_USER_AGENT = YahooCacheSystem
  PATH = /sbin:/bin:/usr/sbin:/usr/bin
  SERVER_SIGNATURE = 
  SERVER_SOFTWARE = Apache/2.2.14
  SERVER_PORT = 80
  REMOTE_ADDR = 98.139.241.249
  REMOTE_PORT = 53863
  GATEWAY_INTERFACE = CGI/1.1
  SERVER_PROTOCOL = HTTP/1.1
  REQUEST_METHOD = GET
  QUERY_STRING = 
  REQUEST_URI = /
  SCRIPT_NAME = /index.php
  PHP_SELF = /index.php
  REQUEST_TIME = 1330923844   
  </code></pre>
  
  <p>Otherwise, is there a test case (suite/simulator) to be able to the test the correct behaviour of a legal bot (only allowing the major ones such as: Google, Yahoo, Bing), to be sure i used the right detection method. There are some simulators around, but most of them are not working properly and the next question is: "can i trust it...".</p>
  
  <p>*Notice: As you can see in the details above, it is using a REMOTE_PORT value of 53863, what kind of port is 53863?*</p>
  
  <p>I hope you can understand my question, if not, drop a line here.</p>