title: Bots throws 500 error in apache access log
body: <p>In my Apache error log I can see the following errors has caught on enormous amount everyday.</p>
  
  <pre><code>[Tue Jan 15 13:37:39 2013] [error] [client 66.249.78.53] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
  </code></pre>
  
  <p>When I check the corroesponding IP, Date and Time with the access log I can see the following </p>
  
  <pre><code>66.249.78.53 - - [15/Jan/2013:13:37:39 +0000] "GET /robots.txt HTTP/1.1" 500 821 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
  </code></pre>
  
  <p>I've tested my robot.txt file in the Google Webmster tool -> Health -> Blocked URLs and it's fine.</p>
  
  <p>Also when some images accessed by bot's it throw the following error,</p>
  
  <p>Error_LOG</p>
  
  <pre><code>[Tue Jan 15 12:14:16 2013] [error] [client 66.249.78.15] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
  </code></pre>
  
  <p>Accessed_URL</p>
  
  <pre><code>66.249.78.15 - - [15/Jan/2013:12:14:16 +0000] "GET /userfiles_generic_imagebank/1335441506.jpg?1 HTTP/1.1" 500 821 "-" "Googlebot-Image/1.0"
  </code></pre>
  
  <p>Actually the above image URL (and several other images in our access log) are not available on our site (they were available before a website revamp that we did in August 2012), and we thrown 404 errors when we go to those invalid resources.</p>
  
  <p>However once in a while, it seems that bots (and even human visitors) generate this type of error in our access/error log, only for static resources like images that don't exist, and our robots.txt file. The server throws a 500 error for them, but actually when I try it from my browser - the images are 404 and the robots.txt is 200 (success).</p>
  
  <p>We are not sure why this is happening and howcome a valid robot.txt and inavalid image can throw a 500 error. We do have a .htaccess file and we are sure that our (Zend framework) application is not being reached, because we have a separate log for that. Therefore, the server itself (or.htaccess) is throwing the 500 error "once in a while" and I can't imagine why. Could it be due to too many requests to the server, or how can I debug this further?</p>
  
  <p>Note that we only noticed these errors after our design revamp, but the web server itself stayed the same</p>