Set audio output device hierarchy?
Try using your Audio MIDI Setup application under the Utilities folder. I don't have multiple outputs to test, but it looks like you may be able to adjust the order of preference.

Also as an alternative workaround you could use a VGA or DVI cable instead of HDMI, which will only transfer video and no audio.