Slow initial response time when website has been idle?
You mention you're using cache, which could mean a whole lot of things, but the symptoms you're describing are typical of what you'd see if you're doing some heavy-handed things inside of a {% cache %}, you let that expire, then you're the first person to hit the page when the cache has to rebuild.

That {% cache %} tag, by default, will purge its cache every 24 hours unless you've specified otherwise.