How to change non-color-related CSS after the color module has been used?
Resave the color settings in the theme; see color_scheme_form_submit().

I'll look into what can be done inside the AdvAgg module: https://www.drupal.org/node/2461039

Edit: I've added this into advagg now. It will regenerate the color css.