Why is bitcoind so slow on VPSs? Sometimes it takes bitcoind more than a minute to respond to basic RPC calls such as getbalance (and on average, it takes at least 5 seconds), even though the "STAT" according to the Linux 'ps' program remains "SLl" (Sleeping!). I've confirmed I was looking at the RPC server process and not the client process. How is this even reasonable?

It shouldn't be possible to run large-scale sites like Mt. Gox with those kind of response times. What special considerations need to be taken into account to get performance out of bitcoind?
bitcoind is often very slow when running over a remote disks (such as on a VPS), especially if that disk is slow.  Check the load of incoming transactions in debug.log, which lock cs_main during processing (which the RPC thread does as well).  If you have the memory, try tmpfs, or increase the bdb cache.