Can Control Panel Settings page become very slow with some plugins?
Is it a reasonable assumption that some plugins may affect Settings page loading time


It's very reasonable.  A plugin can do all sorts of things that might slow down CP rendering and it's reasonable to assume that all plugins don't test with databases that have 25 sections and thousands of entries.

The log files you want to look at are in craft/storage/runtime/logs. craft.log is the most recent and craft.log.1, .2, etc. being older. Do it with devMode enabled and you want to be looking at the timestamps for each entry to see where all of the time is being eaten up.

Alternatively (or in addition to), you can create a Github issue for CacheWarmer and let them know what you're seeing.

Sitemap is commercial, so you can go through the developer's usual support channels.