Is ivy bad for your walls?
Unless my house was all brick and done well I would not let ivy grow on it.  Ivy spreads out and finds its way in any crack.  As the ivy matures it grows thicker to keep spreading.  As it grows thicker it will push out on things once it is in the crack.  I have ivy on a chain link fence in back yard and fence is ruined.

Think about the ivy getting in between the gutters and house.  It just grows so quick during some parts of the year that you might have to potentially check every couple days.

Nonstructural Note:  The bad ivy (poison) can grow with the good.  Taking down the ivy on my back fence resulted in 2 bouts of poison ivy for me.

Obvious Note:  Ivy holds water.  So anything that can decay, rot, warp, or whatever due to water will not last as long.  Also Fiasco Labs has a great point with their "suckers" that they use to attach.  Within a year of freshly painted house you could need a new paint job because of them.