How do I uninstall imagemagick from Mac OS X?
Please try brew remove imagemagick

Sorry my bad ;)
If you manually installed it, goto the root directory where you downloaded the sourcecode and try sudo make uninstall. If you already deleted the install files, redownload the source file and untar it. Then you can cd to the source dir. and use sudo make uninstall to uninstall.