formBuilder()->submitForm() not working
The api documentation for FormBuilder::submitForm is not exactly correct. The $form_arg argument no longer can be a function, but it refers to the class name or form object of the form. In this case ThemeSettingsForm is the class corresponding to system_theme_settings per its getFormId() method return value.

However I am not sure that calling FormBuilder::submitForm will work from your question because form state may not be valid for the system theme settings form. It may be better to track down what specifically color module is requiring in order to set/save the necessary configuration instead of trying to mock form submission.