Once you move beyond 100k hits/month, what would people in this community say is the biggest hurdle?

My situ:
Tons of static media (audio/video/images) being served off of S3/CDN, but being stored locally as backup (not served though). Everything that can be cached, is cached, with about 8 gigs of memory scalable to 32.

We're currently handling about 100k hits without a problem, and would love to know what problems others have run into: load balancing? memory issues? disk i/o?

Thanks for any tips. I've looked through the related questions, and they answered them well, but just wanted to get some more feedback.