Why has Spaces become unbearably slow?
I may not be able to pin down exactly what the bottleneck is, but in effect - OS X tracks every single window opened across all apps and then has to lay them out for each space and decide if a window is visible, occluded by another window or in the dock or hidden entirely by Finder hiding an app.

All of this calculation is pushed to the GPU for buttery smooth animation. At some point, you will exhaust the resources on the GPU whether it's the amount of pixels being stored, the total size of each window, the number of spaces and calculations to be made, transfer speeds from the CPU / RAM to the caches on the GPU. When you hit that bottleneck, either the GPU can't be as efficient or it rejects the work entirely and then you see the CPU doing the rendering.

My suspicion is you are getting the CPU to render which would explain the dramatic drop off in responsiveness. Also - you are pushing somewhere in the order of (5120x2880) + ( 2 x (4096 x 2034) ) = 31,408,128 pixels

If you can make this happen on command - I bet an Apple Engineer would love to look over sudo sysdiagnose WindowServer when the machine is performing miserably. A call to AppleCare or a visit to a genius bar would be the step to get engineering involved.

In the mean time - you might be able to experiment with reducing the load on the GPU by:


less apps with windows open
less windows open per app
fewer virtual screens to contemplate
more hidden windows
re-cabling the displays so they connect directly to the Mac and not daisy chaining them.


For your 4k external displays - are you in MST or SST mode? - https://support.apple.com/en-us/HT206587 - if you went to SST, that might alleviate the bottleneck or require less concessions on the window count at the expense of a 30 Hz refresh rate. The 5k iMac is listed as only supporting one 4k display in MST mode, so that could be an explanation the issue you are seeing.