What is the largest HTC Desire card size?
The desire supports sdhc which is upto 32gb so it should work fine. I have a 16gb in my droid and there's no difference noticable from a smaller card (other than it having more capacity).

If you were to find a bigger card (I'm not sure they have any in micro) you'd need a phone that supports SDXC.