Cached "Page not found" for anonymous users My site's drupal version is 6.19. There are always logs like the following.

the recent log entries shows

type: page not found
location:http://www.example.com/display.html
referrer:
  message: 500.shtml
  severity: warning
  hostname: 95.64.11.29 or others (maybe search engine robot IP)


The URL and the content is right, but when the anonymous user accesses it, it returns the "Page not found" page. Anonymous users have the permissions to access content.

The cache_page table contains the following header for http://www.example.com/display.html:

Content-Type: text/html; charset=utf-8 HTTP/1.1 404 Not Found


Why is the page being cached as "404 Not Found", when it actually exists?

When I access http://www.example.com/display.html as an anonymous user, the site shows "page not found"; when I clear the cache, anonymous users can access the page.
How can I correct the cached content generated by search engine bots?
Is the page actually handled by Drupal or is it a static .html file?
It looks like a file that shouldn't be handled by Drupal, the 500 error 
http://www.checkupdown.com/status/E500.html indicates a fatal server error.

If it is handled by Drupal can you change the url to one without the .html &amp; see if that works with caching. You could use the global_redirect module to forward the .html addresses, once you figure out why it's failing.

If it is a flat file you may need to change the .htaccess or settings.php file to allow the .html to be processed by the webserver without bootstrapping Drupal.

There is a massive thread about this issue on Drupal.org http://drupal.org/node/76824#comment-3299738 You may find an answer in there.

PS 
If this is served on a shared host you may want to contact your provider, they can have some strange setups &amp; can fix quickly if you explain clearly what is happening.