What could cause my house to smell musty after coming back from vacation?
I am guessing you closed all your windows, shut the doors, and made sure the AC was off or set really high.  

So then the humidity of the house goes sky high, mold spores multiply, and all of the major smells in your house that were hidden are multiplied.  I go on a couple of long vacations a year and my house always smells when I get back if it is warm.  Lots of times it leads me to clean something that maybe I didn't notice (steam clean the couch) and sometimes it is me being stupid before I left (leaving dirty clothes out, wet towel on the carpet, left fruit out, half full trash can).  The mixture of no air movement, more moisture, and higher temperatures is an incubator for smell.  And it takes very little mold growth to get that "old people" smell in your house.  Open up all your windows, let the AC chill the place down, and repeat.  Takes a couple days to go back.