Xcode 4.1 not downloading - Lion
You can remove the partial download by searching for Install in spotlight, then return to the purchased section and re-download the installer.

A reboot will ensure it's not paused or some temporary error has caused the slowness.