Can I remove just Music from All My Files?
Given that the results of “All My Files” smart search respects Spotlight Preference Pane's search results, if you uncheck or drag music below pictures, that order is respected.

I infer this, because I have music disabled in Spotlight and I don't see music in my “All My Files”.

I see PDF, Images, Spreadsheets, Developer and Documents (in that order). Which more or less mirrors what I have in my Spotlight: (considering that the canned search doesn't look for contacts, apps, etc.) 



You will need to log off and on again for changes to take effect.

NOTE: I don't use All My Files anyway, I have it removed from my finder Sidebar