Alter Color module fieldset within theme settings form
Definitly system-theme-settings is a core form where color module adding its fields using hook_form_alter in the same manner you are doing.

The point at which your hook_form_alter is invoked seems to be earlier then the color module's.

Check out hook_module_implements_alter to alter the registry of modules implementing a hook.