MacBook Pro to external 24" HP Compaq at 1920x1200 using Mini DisplayPort to VGA adapter
I've had the same problems. 

I think youre leaving the lid to your mac open.

If you are using an external keyboard, external mouse, and an external display, while your mac is powered, you can close the lid, hit the shift key a couple times to wake it up and I think you will see the desired resolution listed. 

There is a command line solution

Get this

After you compile it by running:

./configure

make

sudo make install


you can do 

$ screenresolution get
Display 0: 1280x800x32
$ screenresolution list
Available Modes on Display 0
  1280x800x8    1280x800x16     1280x800x32     1280x800x30 
  1152x720x8    1152x720x16     1152x720x32     1152x720x30 
  1024x768x8    1024x768x16     1024x768x32     1024x768x30 
  1024x768x8    1024x768x16     1024x768x32     1024x768x30 
  1024x640x8    1024x640x16     1024x640x32     1024x640x30 
  800x600x8     800x600x16  800x600x32  800x600x30 
  800x600x8     800x600x16  800x600x32  800x600x30 
  800x500x8     800x500x16  800x500x32  800x500x30 
  640x480x8     640x480x16  640x480x32  640x480x30 
  640x480x8     640x480x16  640x480x32  640x480x30 
  720x480x8     720x480x16  720x480x32  720x480x30 
  720x480x8     720x480x16  720x480x32  720x480x30 

$ screenresolution set 1280x800x32
Setting mode on display 0 to 1280x800x32