Memory-saving cache-clearing strategies for large sites? One of my Drupal 7 sites has thousands of fields, a bunch of content types, more than 25 views, and hundreds (soon to be thousands) of profile types. Because of this, I'm using a core patch that better caches entity field info (http://drupal.org/node/1040790), and the -dev version of Views which better caches views by display (instead of having one HUGE views cache row with all views data in it).

This has helped most pages on the site to load with 20-30MB of RAM used, rather than 160MB+ (instead of pulling up cache_* table rows for fields and views that were 10MB+, the patches help keep cache_* data much more efficient).

This introduces a problem, however, in that cache rebuilds take a really long time. Usually more than a minute or two. And during this time, Drupal simply won't load any pages (since the caches that it's trying to read from are not yet built, other requests have to wait).

During low-traffic cycles, this isn't a big deal; a hundred or so users will simply have to wait a minute before the page loads. But during high-traffic cycles, the Apache server starts going crazy, with 40+ CPU load, and the memory quickly fills up because all the worker threads sit waiting, and max out their memory, causing swapping. It's kind of a death spiral. A restart of httpd will clear things up, but it takes 5-10 minutes for things to get back to normal.

My goal is to make it so cache clears don't bring the site to its knees. For one, if I use admin_menu's individual cache clearing functions (like "CSS and JS", then "Menu", then "Theme registry", etc.), things go smoothly until I hit the "Page and else" option. That's when views' cache is reset (a very CPU and database-intense operation with the number of views that need to be cached), and when the field info cache is reset (which is also CPU and database-intense on this site).

So... my questions/ideas:


Using drush and/or other shell scripting, is it possible for me to clear caches in a more intelligent way than "blast all caches at once, and hope for a clean rebuild"?
Can I block http requests while the cache clearing is happening so apache doesn't get clogged with a bunch of cache-stampeding requests?
If I can clear caches outside of Drupal/normal httpd request, I could presumably set a higher PHP memory_limit for the cache clear operation, and back off my universal memory_limit (right now set to 256MB, in case any individual httpd thread needs to clear caches...).


Basically: Is there any intelligent and graceful way to clear all caches with Drupal besides simply clicking the button in the UI, or using drush cc all?

[Edit for clarification: The main problem I have is cache rebuilds, which (a) take a while, and (b) block all other requests until the rebuilds are complete. I would like to find a way to make it so the rebuilds aren't quite as deadly during high-traffic times.]
Is there any intelligent and graceful way to clear all caches with Drupal besides simply clicking the button in the UI, or using drush cc all?


The cache actions module does that. It depends on rule. For exampple, you can setup a rule to clear a specific view when a node of type "x" has been added or updated. Checkout the docs for more details.

Also take a look at the cache graceful module - haven't tried it yet but looks interesting.