One mini displayport on Radeon HD5870 not working after an OS reinstall; why? (Mac Pro)
I've not had your issue exactly, BUT I've found DVI connection to be the most reliable with external monitors and OS X.  AS you said, you intended to try DVI-D next.  I imagine that would get both monitors firing up.

Side note: OS X 10.8 seems to despise 10.5.  I've had a bunch of unrelated madness crop up due to failed mountain lion install and then recovery via  at some point using 10.5 install disc.  Disastrous results.