How can I prevent some configuration settings getting overwritten?
I believe that Config Ignore could help you out here.

Spoiler: I am the maintainer of that module.

Install the module, go to admin/config/development/configuration/ignore and add the names of the config entities that you would like to not to change, when you import.

In this case:

swiftmailer.transport
system.performance


Save, remember to export config_ignore.settings.yml to the config folder (yes, some of us tend to forget that bit) and you should be golden.