Slow initial response time when website has been idle? I noticed my Craft website is slower to load when it has been idle for awhile. It can take up to 3-4 seconds for the first load, and then it goes down to about 500ms (which is fine) after the first page. I'm also using cache so I could get my query count to about 20-30 per page. I have other websites (non Craft) on the server that are fast to load even when they have been idle for awhile. Any idea what's causing this?
You mention you're using cache, which could mean a whole lot of things, but the symptoms you're describing are typical of what you'd see if you're doing some heavy-handed things inside of a {% cache %}, you let that expire, then you're the first person to hit the page when the cache has to rebuild.

That {% cache %} tag, by default, will purge its cache every 24 hours unless you've specified otherwise.