Reset boost cache in a busy site without crash We are having a high traffic Drupal 6.x site (25,000) uniques per day, mostly landing pages.

Everythings works OK, with great performance, until we try to reset the boost cache. 

Once we reset it, Drupal tries to regenerate HTML from PHP and MYSQL, and we reach a performance crisis where the site crawls and becomes unusable due to slow performance.

Is there a better solution? I imagine there could be a script which clears cache slowly, resets files one by one, wait(10 seconds), wget the url, and then go to the next url.

More info added (edits):


The site has about 20,000 nodes. 
We have two Linode 4096 machines: 4 GB RAM, 8 CPU (4x priority), 192 GB Storage, 8 TB Transfer each. Only one of them stores MySQL.
The reason we do manual purge is because when Boost purges pages by itself, we can get to a point where a lot of pages are not in the cache. This can cause a big performance hit.
When the server is too busy, PHP returns WSOD and then Boost caches it! So we get a page, in cache that is actually WSOD!!! Since this is a critical problem, we decided to affectively disable Boost from purging cache on it own and do it On Demand.
Why clear the boost cache? The CSS and JS files are global. If we change the CSS, or add some JS, then we "drush cc css-js" to load the new changes. Doing that, the CSS/JS files are deleted and we are left with cached HTML pointing to non-existing CSS/JS files. So we get "broken" pages. The solution is to refresh the Boost cache.


Will the advagg module help?

Any other solutions?

Pleae advise.
Here is the script that does the job:

&lt;?php

/* function to list all relevant HTML files in a given directory */
function ListIn($dir, $prefix = '') {
  $dir = rtrim($dir, '\\/');
  $result = array();

    foreach (scandir($dir) as $f) {
      if ($f !== '.' and $f !== '..') {
        if (is_dir("$dir/$f")) {
          $result = array_merge($result, ListIn("$dir/$f", "$prefix$f/"));
        } else {
          if (stripos($f,".gz") === FALSE) {
            $result[] = $prefix.$f;
          }
        }
      }
    }

  return $result;
}

/* main function */
function go() {
  $isToNotify = false;
  // get all relevant HTML files, recursively
  $files = ListIn('/var/www/drupal_site/cache/normal/my.domain.com','');
  foreach($files as $file) {
    // save filename for later
    $filename = str_replace("_.html","",$file);
    // normalize path to file
    $file = "/var/www/drupal_site/cache/normal/my.domain.com/" . $file;
    echo "\n************************************\n\nChecking $file" . "...\n\n";
    // get HTML from file into string variable
    $html = file_get_contents($file);
    // create DOM object for easy parsing
    $dom = new DOMDocument;
    $dom-&gt;loadHTML($html);
    // get all &lt;link&gt; tags
    $links = $dom-&gt;getElementsByTagName("link");
    foreach($links as $link) {
      $css = $link-&gt;getAttribute('href');
      $css = str_replace("http://my.domain.com","",$css);
      // find only the relevant tags with the CSS file
      if (stripos($css,"css_") !== FALSE) {
        // normalize path to CSS file
        $path = "/var/www/drupal_site" .$css;
        // check if CSS exists
        if (file_exists($path)) {
          echo "The file $path exists\n";
          // no need to do anything else
        }
        else {
          echo "The file $path does not exist\n";
          // set notification ON
          $isToNotify = true;
          // delete HTML with links to not existing CSS file
          unlink("/var/www/drupal_site/cache/normal/my.domain.com/$filename"."_.html");
          unlink("/var/www/drupal_site/cache/normal/my.domain.com/$filename"."_.html.gz");
          // GET the HTML from scratch so Boost will re-cache it
          echo file_get_contents("http://my.domain.com/$filename");
          echo "Adding log entry...\n";
          // log activity
          $content = "\n\n".(date("Y-m-d H:i:s"))."\n    Refreshed HTML: my.domain.com/".$filename."\n    CSS file which was not found: ".$css;
          file_put_contents("/root/scripts/logs/monitorCSS.log",$content,FILE_APPEND);
          echo "Going to sleep for a while ZzZzZz...\n";
          // sleep for a while to avoid bombardment of the server by this script
          sleep(20); // in seconds
          // go to next file - no need to refresh same file more than once (if more than one CSS file was not-found in HTML)
          break;
        }
      }
    }
  }
  if ($isToNotify) {
    // send email notification if needed (we found some missing CSS files)
    mail('my@email.address', 'Monitor CSS Notification', wordwrap("Some missing CSS files discovered - see log file for more info...", 70));
  }
}

// call main function
go();
?&gt;