Why is the first visit to a rarely used site slow or even lead to timeout?
There isn't a lot of information, and the answer is likely specific to your instance, but my first guess is that you are experiencing the effects of one or more cache misses.

If you are using APC, then your rarely used site may not have core files in the bytecode cache.  Subsequent visits will.

If you are using MySQL query caching, then your rarely used site likely doesn't have anything in the cache, so all queries need to actually run.

The Drupal cache entries themselves (ie, the variouls cache table) may also be stale, so initial slowdowns could be caused by this.

If you are using Poormanscron, then this could be running, too, on initial page loads.