Day: February 21, 2006

Freezing in SLT

President’s Day weekend, and the slopes were packed. I had decided to go with a friend from L.A. who had never been skiing or boarding before, so a lot of the time was spent outside the ski circus. In the end, I had a ton of fun, although the conditions were only average.

The snow was decent, with some four inches falling between Friday and Saturday. Sunday saw a few flurries, but nothing that amounted to what the other resorts had seen. Mt. Rose surprised us with over a foot of fresh snow! Oddly enough, the Nevada side had much better conditions. The top of Galaxy had some serious powder cover, and I had a great time sailing through it.

The good side of the poor snow was that a lot of people had deserted Heavenly and gone to the North Shore. Northstar must have been drowned, just as Mt. Rose or Squaw. We had a good time – nothing of the 45 minute wait of last year, and even when there were lines, they usually were only at one of two lifts (e.g. at Sky Express, but not at Canyon Express).


Recovering from IMAP crash

One morning I connect to my IMAP server and it tells me there is an I/O error. Terrified, I connect using SSH and find out the hard drive is full of backup files. This in turn seems to have caught Cyrus in the worst possible moment, and I struggle to restart it. Whenever I do, I get weird error messages like this:
Feb 21 04:35:15 invent lmtpunix[9573]: DBERROR db4: PANIC: fatal region error detected; run recovery
Feb 21 04:35:15 invent lmtpunix[9573]: DBERROR: critical database situation
Of course, I have not the slightest idea what to do. At least there is a hint to a recovery tool, no?

What To Look At

First things first: the Cyrus documentation is not as good as it should be, but not as bad as it could. Fortunately, a lot of people seem to have had similar problems, and I could garner a lot of information.

There are four places that store information relevant to this process:

  1. /var/spool/imap the sacred location of the mail store (your email)
  2. /usr/lib/cyrus where the tools for cyrus are located
  3. /etc with the configuration files
  4. /var/lib/imap the location of the corrupted db files

I had no idea what was corrupt, but got good vibes running ‘file’ on most of the db files in the last directory. Only the mailboxes.db file and the db directory seemed out of whack. That’s good, you can rebuild both easily (if you know how).

First Things First

Back up the directories listed above, at the very least the /var/spool/imap and /var/lib/imap directories! You are going to do some really invasive surgery around here, and you want to be sure you don’t lose anything permanently!

The Script

There is a trick – you have to run a script that does most of the work for you. It is listed under "More…". Copy and paste it into an editor, save it under a convenient name, chmod +x it, and run. It should do all its work without much complaining. I got it from a Google source, that had a bunch of things wrong. After fixing paths and adding the right secret sauce, it worked just fine.

To run the script, first shut down the IMAP server, then execute. Once the script is done (which took about 10 minutes on my machine), you can restart the server.

Ensure you read the output of /var/log/mail and /var/log/messages to ensure everything is fine! The first few times, I had gotten it wrong, and the log files quickly grew again into making my system run out of space.