Drupal is slow only in IE
Javascript is probably your villain.

Especially on IE6 and IE7 and still in IE8, Javascript is MUCH slower than the rest of the world.

Despite not being the most brilliant piece of js ever, drupal.js is harmless.  The real harm probably lies a lot of plugins or maybe one or two poorly-coded scripts that perform some kind of expensive DOM lookup or other task that is negligeable in Firefox et al but really slows down IE.