How important are tones in comparison with vowels or consonants in Mandarin?
Mispronunciation of tones sounds "more wrong", probably not because it is more wrong, but rather more random and so, harder to determine what the speaker means to say. Mispronunciation of sounds tends to be systematic, e.g. consistently mispronouncing "zh" as "z", etc. This is called an accent.
Every language that we know of has consonant and vowel sounds. When one tries to learn a foreign language with unfamiliar sounds, one may substitute, thus resulting in an accent. Most people can understand someone who speaks with an accent because as I said above, it is a systematic substitution of one sound for another.

Tones, however, presents quite a different problem for learners whose native language does not have tones. The mistakes are likely more random, and so, harder for the hearer to decipher what the intended tone is. And because of this, it sounds "more wrong".

Having said this, however, for learners whose native language has tones, for instance, a Cantonese speaker learning Mandarin, or the other way around, then the mistake will again likely be more systematic. 

A person who is learning Chinese should really think of the consonant/vowel/tone as a packaged deal. All components are essential to pronouncing a word correctly.

Happy learning!