Why is the first visit to a rarely used site slow or even lead to timeout? I am looking for suggestions on Apache/Drupal/MySQL (localhost) configuration, to solve my issue. I use Ubuntu servers 8 LTS with classic Apache installation, and perform manual Drupal installation too. After publishing a site for customer review, I notice, that if a site is not visited very often or not visited at all for some time, such site response time increases significantly, up to situations with timeouts. Although, other vhost sites not on drupal on the same server run smoothly. E.g., if a site is rarely used, opening and navigating through out it takes very long times, or even timeouts. Each next page visit opens faster.

I use regular Apache settings and PHP memory up to 512MB, as sometimes Drupal module installations ask for more. Server is with 4GB RAM, and around 10 sites running on it at a time. Drupal for customer review is set without caching and performance settings turned on, thus I consider, that this can be a bottle neck in some cases, but still, if I work on the site, it is quite responsive, even without performance settings turned on.

I understand that there are very many ways of solutions and potential issue roots, but wondering, if there are some constructive ways to track this down to small list of tasks to accomplish better configuration. I've experimented with different Drupal performance tuning solutions found on the net, but it still does not solve issue, if a site is not used frequently. Restarting apache helps for a while.
There isn't a lot of information, and the answer is likely specific to your instance, but my first guess is that you are experiencing the effects of one or more cache misses.

If you are using APC, then your rarely used site may not have core files in the bytecode cache.  Subsequent visits will.

If you are using MySQL query caching, then your rarely used site likely doesn't have anything in the cache, so all queries need to actually run.

The Drupal cache entries themselves (ie, the variouls cache table) may also be stale, so initial slowdowns could be caused by this.

If you are using Poormanscron, then this could be running, too, on initial page loads.